S3へ出力したCloudWatch LogsのデータをS3 SelectとAthenaで確認してみた
Kinesis Data Firehose介し、S3へ出力したCloudWatch LogsのデータをS3 SelectとAthenaで確認してみました。
Kinesis Data Firehoseを使いCloudWatch LogのデータをS3に出力することが可能です。S3をデータソースにしたデータの確認方法は多々あると思いますが、ここではS3 SelectとAthenaを利用しログデータを確認してみたいと思います。本エントリでは、環境構築については割愛していますので、構築については以下ブログを参考にしてください。
なお、ここではS3へ出力されたログデータはAurora監査ログを利用しています。
S3 Select
構成
Kinesis Data Firehose設定
配信ストリームの設定は以下となります。
配信ストリーム設定
$ aws firehose describe-delivery-stream \
--delivery-stream-name delivery-stream-for-s3-select
{
"DeliveryStreamDescription": {
"DeliveryStreamName": "delivery-stream-for-s3-select",
"DeliveryStreamARN": "arn:aws:firehose:ap-northeast-1:XXXXXXXXXXXX:deliverystream/delivery-stream-for-s3-select",
"DeliveryStreamStatus": "ACTIVE",
"DeliveryStreamEncryptionConfiguration": {
"Status": "DISABLED"
},
"DeliveryStreamType": "DirectPut",
"VersionId": "1",
"CreateTimestamp": 1588932218.145,
"Destinations": [
{
"DestinationId": "destinationId-000000000001",
"S3DestinationDescription": {
"RoleARN": "arn:aws:iam::XXXXXXXXXXXX:role/TestFirehosetoS3Role",
"BucketARN": "arn:aws:s3:::cloudwatch-logs-for-s3-select",
"Prefix": "test-aurora-cluster/audit/",
"ErrorOutputPrefix": "error-test-aurora-cluster/audit/",
"BufferingHints": {
"SizeInMBs": 1,
"IntervalInSeconds": 60
},
"CompressionFormat": "UNCOMPRESSED",
"EncryptionConfiguration": {
"NoEncryptionConfig": "NoEncryption"
},
"CloudWatchLoggingOptions": {
"Enabled": true,
"LogGroupName": "/aws/kinesisfirehose/delivery-stream-for-s3-select",
"LogStreamName": "S3Delivery"
}
},
"ExtendedS3DestinationDescription": {
"RoleARN": "arn:aws:iam::XXXXXXXXXXXX:role/TestFirehosetoS3Role",
"BucketARN": "arn:aws:s3:::cloudwatch-logs-for-s3-select",
"Prefix": "test-aurora-cluster/audit/",
"ErrorOutputPrefix": "error-test-aurora-cluster/audit/",
"BufferingHints": {
"SizeInMBs": 1,
"IntervalInSeconds": 60
},
"CompressionFormat": "UNCOMPRESSED",
"EncryptionConfiguration": {
"NoEncryptionConfig": "NoEncryption"
},
"CloudWatchLoggingOptions": {
"Enabled": true,
"LogGroupName": "/aws/kinesisfirehose/delivery-stream-for-s3-select",
"LogStreamName": "S3Delivery"
},
"ProcessingConfiguration": {
"Enabled": false,
"Processors": []
},
"S3BackupMode": "Disabled"
}
}
],
"HasMoreDestinations": false
}
}
CloudWatch Logsサブスクリプションフィルタ設定
サブスクリプションフィルタの設定は以下となります。
サブスクリプションフィルタ設定
$ aws logs describe-subscription-filters \
--log-group-name /aws/rds/cluster/test-aurora-cluster/audit
{
"subscriptionFilters": [
{
"filterName": "Destination",
"logGroupName": "/aws/rds/cluster/test-aurora-cluster/audit",
"filterPattern": "",
"destinationArn": "arn:aws:firehose:ap-northeast-1:XXXXXXXXXXXX:deliverystream/delivery-stream-for-s3-select",
"roleArn": "arn:aws:iam::XXXXXXXXXXXX:role/TestCWLtoKinesisFirehoseRole",
"distribution": "ByLogStream",
"creationTime": 1588816544843
}
]
}
CloudWatch Logロギング
後ほどログの確認が行いやすいよう、AuroraにアクセスしCloudWatch Logへロギングします。
Auroraでのオペレーション
MySQL [(none)]> select now(); +---------------------+ | now() | +---------------------+ | 2020-05-08 19:33:52 | +---------------------+ 1 row in set (0.00 sec) MySQL [(none)]> select user(); +-----------------+ | user() | +-----------------+ | [email protected] | +-----------------+ 1 row in set (0.00 sec) MySQL [(none)]> select host,user,authentication_string from mysql.user; +-----------+-----------+-------------------------------------------+ | host | user | authentication_string | +-----------+-----------+-------------------------------------------+ | % | admin | *2470C0C06DEE42FD1618BB99005ADCA2EC9D1E19 | | % | test_user | *2470C0C06DEE42FD1618BB99005ADCA2EC9D1E19 | | % | testadmin | *2470C0C06DEE42FD1618BB99005ADCA2EC9D1E19 | | localhost | mysql.sys | *THISISNOTAVALIDPASSWORDTHATCANBEUSEDHERE | | localhost | rdsadmin | *8C0E64674BB3F606FF339142F68298EB1061E446 | +-----------+-----------+-------------------------------------------+ 5 rows in set (0.00 sec) MySQL [(none)]> select now(); +---------------------+ | now() | +---------------------+ | 2020-05-08 19:33:52 | +---------------------+ 1 row in set (0.00 sec)
しばらくすると、CloudWatch Logへのロギングが確認できました。
CloudWatch Logロギング確認
$ QUERY_ID=`aws logs start-query \
--log-group-name '/aws/rds/cluster/test-aurora-cluster/audit' \
--start-time 1588934030 \
--end-time 1588934040 \
--query-string \
'parse "*,*,*,*,*,*,*,*,*,*" as timestamp,serverhost,username,host,connectionid,queryid,operation,database,object,retcode
| filter username = "admin"
| sort timestamp asc' \
--output text`
$ aws logs get-query-results \
--query-id ${QUERY_ID} \
--output table
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
| GetQueryResults |
+-------------------------------------------------------------------------------------+--------------------------------------------------------------------------------------------------------+
| status | Complete |
+-------------------------------------------------------------------------------------+--------------------------------------------------------------------------------------------------------+
|| results ||
|+--------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+|
|| field | value ||
|+--------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+|
|| timestamp | 1588934032041466 ||
|| serverhost | test-aurora-master-instance ||
|| username | admin ||
|| host | 10.0.1.54 ||
|| connectionid| 40 ||
|| queryid | 3801 ||
|| operation | QUERY ||
|| database | ||
|| object | 'select now()' ||
|| retcode | 0 ||
|| @ptr | CnYKOwo3MzkyMTM4Nzg5NzY0Oi9hd3MvcmRzL2NsdXN0ZXIvdGVzdC1hdXJvcmEtY2x1c3Rlci9hdWRpdBAGEjcaGAIF5hMtKQAAAAE3eR+gAAXrU1gAAAAAkiABKNCS5J6fLjCC5u2eny44gAFAy6QBSOlBUIw+EBYYAQ== ||
|| timestamp | 1588934032042572 ||
|| serverhost | test-aurora-master-instance ||
|| username | admin ||
|| host | 10.0.1.54 ||
|| connectionid| 40 ||
|| queryid | 3802 ||
|| operation | QUERY ||
|| database | ||
|| object | 'select user()' ||
|| retcode | 0 ||
|| @ptr | CnYKOwo3MzkyMTM4Nzg5NzY0Oi9hd3MvcmRzL2NsdXN0ZXIvdGVzdC1hdXJvcmEtY2x1c3Rlci9hdWRpdBAGEjcaGAIF5hMtKQAAAAE3eR+gAAXrU1gAAAAAkiABKNCS5J6fLjCC5u2eny44gAFAy6QBSOlBUIw+EBcYAQ== ||
|| timestamp | 1588934032044153 ||
|| serverhost | test-aurora-master-instance ||
|| username | admin ||
|| host | 10.0.1.54 ||
|| connectionid| 40 ||
|| queryid | 3803 ||
|| operation | READ ||
|| database | mysql ||
|| object | user ||
|| retcode | ||
|| @ptr | CnYKOwo3MzkyMTM4Nzg5NzY0Oi9hd3MvcmRzL2NsdXN0ZXIvdGVzdC1hdXJvcmEtY2x1c3Rlci9hdWRpdBAGEjcaGAIF5hMtKQAAAAE3eR+gAAXrU1gAAAAAkiABKNCS5J6fLjCC5u2eny44gAFAy6QBSOlBUIw+EBkYAQ== ||
|| timestamp | 1588934032044264 ||
|| serverhost | test-aurora-master-instance ||
|| username | admin ||
|| host | 10.0.1.54 ||
|| connectionid| 40 ||
|| queryid | 3803 ||
|| operation | QUERY ||
|| database | mysql ||
|| object | 'select host ||
|| retcode | user,authentication_string from mysql.user',0 ||
|| @ptr | CnYKOwo3MzkyMTM4Nzg5NzY0Oi9hd3MvcmRzL2NsdXN0ZXIvdGVzdC1hdXJvcmEtY2x1c3Rlci9hdWRpdBAGEjcaGAIF5hMtKQAAAAE3eR+gAAXrU1gAAAAAkiABKNCS5J6fLjCC5u2eny44gAFAy6QBSOlBUIw+EBoYAQ== ||
|| timestamp | 1588934032788148 ||
|| serverhost | test-aurora-master-instance ||
|| username | admin ||
|| host | 10.0.1.54 ||
|| connectionid| 40 ||
|| queryid | 3807 ||
|| operation | QUERY ||
|| database | mysql ||
|| object | 'select now()' ||
|| retcode | 0 ||
|| @ptr | CnUKOwo3MzkyMTM4Nzg5NzY0Oi9hd3MvcmRzL2NsdXN0ZXIvdGVzdC1hdXJvcmEtY2x1c3Rlci9hdWRpdBAFEjYaGAIF55NVCwAAAAF4y6Z3AAXrU1VwAAABsiABKIG74Z6fLjDXoeueny44c0CHpAFIjUhQs0QQKhgB ||
|+--------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+|
|| statistics ||
|+--------------------------------------------------------+-----------------------------------------------------------------+-----------------------------------------------------------------+|
|| bytesScanned | recordsMatched | recordsScanned ||
|+--------------------------------------------------------+-----------------------------------------------------------------+-----------------------------------------------------------------+|
|| 178911.0 | 5.0 | 1027.0 ||
|+--------------------------------------------------------+-----------------------------------------------------------------+-----------------------------------------------------------------+|
S3 Select実行
S3 SelectはS3に格納された単一のオブジェクトより、SQLにてデータを抽出することが可能です。さきほど実施した、Auroraでのオペレーションは、JSTの「2020-05-08 19:33」頃となります。今回の配信ストリームの設定では、「YYYY/MM/DD/HH（UTC）」形式でS3にデータが出力されます。出力先のバケット設定などを加味すると、以下が出力先となります。
- s3://cloudwatch-logs-for-s3-select/test-aurora-cluster/audit/2020/05/08/10
更新日時を参考に、S3 Selectを行うログをピックアップします。
「アクション」-「S3 Select」をクリックします。
CloudWatch LogsからKinesis Data Firehoseを介しS3に出力されたデータは、JSON形式でgzipに圧縮されていますので、以下を選択し「ファイルプレビューの表示」をクリックします。
- ファイル形式…JSON
- JSONタイプ…JSON ドキュメント
- 圧縮…GZIP
この時確認できたプレビューデータを添付しておきます。
プレビュー結果
[
{
"messageType": "DATA_MESSAGE",
"owner": "XXXXXXXXXXXX",
"logGroup": "/aws/rds/cluster/test-aurora-cluster/audit",
"logStream": "test-aurora-master-instance.audit.log.3.2020-05-08-10-22.0.1",
"subscriptionFilters": [
"Destination"
],
"logEvents": [
{
"id": "35434412931205709067612018357150400014362387976194490368",
"timestamp": 1588934029592,
"message": "1588934029592349,test-aurora-master-instance,rdsadmin,localhost,3,3791,QUERY,mysql,'SET @@sql_log_bin=on',0"
},
{
"id": "35434412941218743661752268147699937518781502292379697153",
"timestamp": 1588934030041,
"message": "1588934030041805,test-aurora-master-instance,rdsadmin,localhost,2,3792,QUERY,mysql,'set local oscar_local_only_replica_host_status=1',0"
},
{
"id": "35434412941241044406950798770841473237054150653885677570",
"timestamp": 1588934030042,
"message": "1588934030042281,test-aurora-master-instance,rdsadmin,localhost,2,3793,QUERY,mysql,'SELECT durable_lsn, current_read_point, server_id, last_update_timestamp FROM information_schema.replica_host_status',0"
},
{
"id": "35434412941285645897347860017124544673599447376897638403",
"timestamp": 1588934030044,
"message": "1588934030044855,test-aurora-master-instance,rdsadmin,localhost,2,3794,QUERY,mysql,'set local oscar_local_only_replica_host_status=0',0"
}
]
},
{
"messageType": "DATA_MESSAGE",
"owner": "XXXXXXXXXXXX",
"logGroup": "/aws/rds/cluster/test-aurora-cluster/audit",
"logStream": "test-aurora-master-instance.audit.log.2.2020-05-08-10-22.0.3",
"subscriptionFilters": [
"Destination"
],
"logEvents": [
{
"id": "35434412929800762120104589099274971885390409568918831104",
"timestamp": 1588934029529,
"message": "1588934029529190,test-aurora-master-instance,rdsadmin,localhost,3,3789,QUERY,mysql,'select @@session.tx_read_only',0"
}
]
},
{
"messageType": "DATA_MESSAGE",
"owner": "XXXXXXXXXXXX",
"logGroup": "/aws/rds/cluster/test-aurora-cluster/audit",
"logStream": "test-aurora-master-instance.audit.log.0.2020-05-08-10-22.0.3",
"subscriptionFilters": [
"Destination"
],
"logEvents": [
{
"id": "35434412927548386855052996162013805543482168054050848768",
"timestamp": 1588934029428,
"message": "1588934029428140,test-aurora-master-instance,rdsadmin,localhost,3,3786,QUERY,mysql,'SET @@sql_log_bin=off',0"
},
{
"id": "35434412928551920388986874203382912865751344321819967489",
"timestamp": 1588934029473,
"message": "1588934029473859,test-aurora-master-instance,rdsadmin,localhost,3,3788,WRITE,mysql,rds_heartbeat2,"
}
]
},
{
"messageType": "DATA_MESSAGE",
"owner": "XXXXXXXXXXXX",
"logGroup": "/aws/rds/cluster/test-aurora-cluster/audit",
"logStream": "test-aurora-master-instance.audit.log.1.2020-05-08-10-22.0.3",
"subscriptionFilters": [
"Destination"
],
"logEvents": [
{
"id": "35434412928440416662994221087716244297993433824790904832",
"timestamp": 1588934029468,
"message": "1588934029468081,test-aurora-master-instance,rdsadmin,localhost,3,3787,QUERY,mysql,'select @@session.tx_read_only',0"
},
{
"id": "35434412929510852432523690998509958775080555177077964801",
"timestamp": 1588934029516,
"message": "1588934029516681,test-aurora-master-instance,rdsadmin,localhost,3,3788,QUERY,mysql,'INSERT INTO mysql.rds_heartbeat2(id, value) values (1,1588934029428) ON DUPLICATE KEY UPDATE value = 1588934029428',0"
},
{
"id": "35434412930670491182847283401869816125258269975388946434",
"timestamp": 1588934029568,
"message": "1588934029568797,test-aurora-master-instance,rdsadmin,localhost,3,3790,QUERY,mysql,'COMMIT',0"
}
]
},
{
"messageType": "DATA_MESSAGE",
"owner": "XXXXXXXXXXXX",
"logGroup": "/aws/rds/cluster/test-aurora-cluster/audit",
"logStream": "test-aurora-master-instance.audit.log.1.2020-05-08-10-22.0.3",
"subscriptionFilters": [
"Destination"
],
"logEvents": [
{
"id": "35434412971859967564533344346683415762524399170967961600",
"timestamp": 1588934031415,
"message": "1588934031415444,test-aurora-master-instance,rdsadmin,localhost,3,3795,QUERY,mysql,'SET @@sql_log_bin=off',0"
}
]
},
{
"messageType": "DATA_MESSAGE",
"owner": "XXXXXXXXXXXX",
"logGroup": "/aws/rds/cluster/test-aurora-cluster/audit",
"logStream": "test-aurora-master-instance.audit.log.0.2020-05-08-10-22.0.0",
"subscriptionFilters": [
"Destination"
],
"logEvents": [
{
"id": "35434412973465621218827549212830711225717307494644121600",
"timestamp": 1588934031487,
"message": "1588934031487938,test-aurora-master-instance,rdsadmin,localhost,3,3796,QUERY,mysql,'select @@session.tx_read_only',0"
},
{
"id": "35434412973487921964026079835972246943989955856150102017",
"timestamp": 1588934031488,
"message": "1588934031488740,test-aurora-master-instance,rdsadmin,localhost,3,3797,WRITE,mysql,rds_heartbeat2,"
},
{
"id": "35434412974469154752761427254199818547986483762413240322",
"timestamp": 1588934031532,
"message": "1588934031532851,test-aurora-master-instance,rdsadmin,localhost,3,3798,QUERY,mysql,'select @@session.tx_read_only',0"
},
{
"id": "35434412975717996483879142150125818771254792006748143619",
"timestamp": 1588934031588,
"message": "1588934031588794,test-aurora-master-instance,rdsadmin,localhost,3,3799,QUERY,mysql,'COMMIT',0"
},
{
"id": "35434412985820234058813514433241499148764499768957272068",
"timestamp": 1588934032041,
"message": "1588934032041466,test-aurora-master-instance,admin,10.0.1.54,40,3801,QUERY,,'select now()',0"
},
{
"id": "35434412985842534804012045056383034867037148130463252485",
"timestamp": 1588934032042,
"message": "1588934032042572,test-aurora-master-instance,admin,10.0.1.54,40,3802,QUERY,,'select user()',0"
},
{
"id": "35434412985887136294409106302666106303582444853475213318",
"timestamp": 1588934032044,
"message": "1588934032044129,test-aurora-master-instance,rdsadmin,localhost,2,3804,QUERY,mysql,'set local oscar_local_only_replica_host_status=1',0"
},
{
"id": "35434412985887136294409106302666106303582444853475213319",
"timestamp": 1588934032044,
"message": "1588934032044153,test-aurora-master-instance,admin,10.0.1.54,40,3803,READ,mysql,user,"
},
{
"id": "35434412985887136294409106302666106303582444853475213320",
"timestamp": 1588934032044,
"message": "1588934032044264,test-aurora-master-instance,admin,10.0.1.54,40,3803,QUERY,mysql,'select host,user,authentication_string from mysql.user',0"
},
{
"id": "35434412985887136294409106302666106303582444853475213321",
"timestamp": 1588934032044,
"message": "1588934032044753,test-aurora-master-instance,rdsadmin,localhost,2,3805,QUERY,mysql,'SELECT durable_lsn, current_read_point, server_id, last_update_timestamp FROM information_schema.replica_host_status',0"
},
{
"id": "35434412985909437039607636925807642021855093214981193738",
"timestamp": 1588934032045,
"message": "1588934032045354,test-aurora-master-instance,rdsadmin,localhost,2,3806,QUERY,mysql,'set local oscar_local_only_replica_host_status=0',0"
}
]
},
{
"messageType": "DATA_MESSAGE",
"owner": "XXXXXXXXXXXX",
"logGroup": "/aws/rds/cluster/test-aurora-cluster/audit",
"logStream": "test-aurora-master-instance.audit.log.3.2020-05-08-10-22.0.2",
"subscriptionFilters": [
"Destination"
],
"logEvents": [
{
"id": "35434412973710929416011386067329825380935958304238665728",
"timestamp": 1588934031498,
"message": "1588934031498452,test-aurora-master-instance,rdsadmin,localhost,3,3797,QUERY,mysql,'INSERT INTO mysql.rds_heartbeat2(id, value) values (1,1588934031415) ON DUPLICATE KEY UPDATE value = 1588934031415',0"
},
{
"id": "35434412976164011387849754612898754390927278069896511489",
"timestamp": 1588934031608,
"message": "1588934031608371,test-aurora-master-instance,rdsadmin,localhost,3,3800,QUERY,mysql,'SET @@sql_log_bin=on',0"
}
]
},
{
"messageType": "DATA_MESSAGE",
"owner": "XXXXXXXXXXXX",
"logGroup": "/aws/rds/cluster/test-aurora-cluster/audit",
"logStream": "test-aurora-master-instance.audit.log.2.2020-05-08-10-22.0.2",
"subscriptionFilters": [
"Destination"
],
"logEvents": [
{
"id": "35434413002478890722115889921126861524238751421182902272",
"timestamp": 1588934032788,
"message": "1588934032788148,test-aurora-master-instance,admin,10.0.1.54,40,3807,QUERY,mysql,'select now()',0"
}
]
},
{
"messageType": "DATA_MESSAGE",
"owner": "XXXXXXXXXXXX",
"logGroup": "/aws/rds/cluster/test-aurora-cluster/audit",
"logStream": "test-aurora-master-instance.audit.log.3.2020-05-08-10-22.0.2",
"subscriptionFilters": [
"Destination"
],
"logEvents": [
{
"id": "35434413018535427265057938584227977259756395433104179200",
"timestamp": 1588934033508,
"message": "1588934033508241,test-aurora-master-instance,rdsadmin,localhost,3,3810,QUERY,mysql,'INSERT INTO mysql.rds_heartbeat2(id, value) values (1,1588934033436) ON DUPLICATE KEY UPDATE value = 1588934033436',0"
}
]
},
{
"messageType": "DATA_MESSAGE",
"owner": "XXXXXXXXXXXX",
"logGroup": "/aws/rds/cluster/test-aurora-cluster/audit",
"logStream": "test-aurora-master-instance.audit.log.0.2020-05-08-10-22.0.1",
"subscriptionFilters": [
"Destination"
],
"logEvents": [
{
"id": "35434413016907472865565203094980709992037281047195680768",
"timestamp": 1588934033435,
"message": "1588934033435781,test-aurora-master-instance,rdsadmin,localhost,3,3808,QUERY,mysql,'SET @@sql_log_bin=off',0"
},
{
"id": "35434413019538960798991816625681924748209787704901369857",
"timestamp": 1588934033553,
"message": "1588934033553002,test-aurora-master-instance,rdsadmin,localhost,3,3812,QUERY,mysql,'COMMIT',0"
},
{
"id": "35434413020430990606933041551343353479115722165140586498",
"timestamp": 1588934033593,
"message": "1588934033593076,test-aurora-master-instance,rdsadmin,localhost,3,3813,QUERY,mysql,'SET @@sql_log_bin=on',0"
}
]
}
]
SQLエディタにて、以下SQLを入力し「SQLの実行」をクリックします。
select * from S3Object[*].logEvents[*].message s where s like '%,admin%'
クエリについて補足します。テーブルS3Objectを指定し、JSONオブジェクト内の配列、キーを指定しています。（ここでは、logEvents配列内のmessageキー）条件は文字列の部分一致で、さきほどAuroraでオペレーションを行ったユーザadminを指定しました。詳細については以下を確認ください。
SQLの結果は以下となり、さきほど実施したAuroraでのオペレーションをログから確認することができました。
[
{
"_1": "1588934032041466,test-aurora-master-instance,admin,10.0.1.54,40,3801,QUERY,,'select now()',0"
},
{
"_1": "1588934032042572,test-aurora-master-instance,admin,10.0.1.54,40,3802,QUERY,,'select user()',0"
},
{
"_1": "1588934032044153,test-aurora-master-instance,admin,10.0.1.54,40,3803,READ,mysql,user,"
},
{
"_1": "1588934032044264,test-aurora-master-instance,admin,10.0.1.54,40,3803,QUERY,mysql,'select host,user,authentication_string from mysql.user',0"
},
{
"_1": "1588934032788148,test-aurora-master-instance,admin,10.0.1.54,40,3807,QUERY,mysql,'select now()',0"
},
{
"_1": "1588934043462741,test-aurora-master-instance,admin,10.0.1.54,40,0,DISCONNECT,,,0"
}
]
AWS CLIで実施する場合は以下のようになります。
$ aws s3api select-object-content \
--bucket=cloudwatch-logs-for-s3-select \
--key=test-aurora-cluster/audit/2020/05/08/10/delivery-stream-for-s3-select-1-2020-05-08-10-33-51-8bf316ce-efd0-47e9-ba9d-57726e11cd5f \
--expression "select * from S3Object[*].logEvents[*].message s where s like '%,admin%'" \
--input-serialization '{ "CompressionType": "GZIP","JSON": {"Type": "DOCUMENT"}}' \
--output-serialization '{"JSON":{"RecordDelimiter":"\n"}}' \
--expression-type SQL output.json
{"_1":"1588934032041466,test-aurora-master-instance,admin,10.0.1.54,40,3801,QUERY,,'select now()',0"}
{"_1":"1588934032042572,test-aurora-master-instance,admin,10.0.1.54,40,3802,QUERY,,'select user()',0"}
{"_1":"1588934032044153,test-aurora-master-instance,admin,10.0.1.54,40,3803,READ,mysql,user,"}
{"_1":"1588934032044264,test-aurora-master-instance,admin,10.0.1.54,40,3803,QUERY,mysql,'select host,user,authentication_string from mysql.user',0"}
{"_1":"1588934032788148,test-aurora-master-instance,admin,10.0.1.54,40,3807,QUERY,mysql,'select now()',0"}
{"_1":"1588934043462741,test-aurora-master-instance,admin,10.0.1.54,40,0,DISCONNECT,,,0"}
Athena
構成
Kinesis Data Firehoseを介し、S3に出力されたCloudWatch Logsのデータは以下のように、1行に複数のJSONが保存されています。
$ aws s3 cp s3://cloudwatch-logs-for-s3-select/test-aurora-cluster/audit/2020/05/08/10/delivery-stream-for-s3-select-1-2020-05-08-10-33-51-8bf316ce-efd0-47e9-ba9d-57726e11cd5f ./cwl.gz
download: s3://cloudwatch-logs-for-s3-select/test-aurora-cluster/audit/2020/05/08/10/delivery-stream-for-s3-select-1-2020-05-08-10-33-51-8bf316ce-efd0-47e9-ba9d-57726e11cd5f to ./cwl.gz
$ gzcat cwl.gz
{"messageType":"DATA_MESSAGE","owner":"XXXXXXXXXXXX","logGroup":"/aws/rds/cluster/test-aurora-cluster/audit","logStream":"test-aurora-master-instance.audit.log.3.2020-05-08-10-22.0.1","subscriptionFilters":["Destination"],"logEvents":[{"id":"35434412931205709067612018357150400014362387976194490368","timestamp":1588934029592,"message":"1588934029592349,test-aurora-master-instance,rdsadmin,localhost,3,3791,QUERY,mysql,'SET @@sql_log_bin=on',0"},{"id":"35434412941218743661752268147699937518781502292379697153","timestamp":1588934030041,"message":"1588934030041805,test-aurora-master-instance,rdsadmin,localhost,2,3792,QUERY,mysql,'set local oscar_local_only_replica_host_status=1',0"},{"id":"35434412941241044406950798770841473237054150653885677570","timestamp":1588934030042,"message":"1588934030042281,test-aurora-master-instance,rdsadmin,localhost,2,3793,QUERY,mysql,'SELECT durable_lsn, current_read_point, server_id, last_update_timestamp FROM information_schema.replica_host_status',0"},{"id":"35434412941285645897347860017124544673599447376897638403","timestamp":1588934030044,"message":"1588934030044855,test-aurora-master-instance,rdsadmin,localhost,2,3794,QUERY,mysql,'set local oscar_local_only_replica_host_status=0',0"}]}{"messageType":"DATA_MESSAGE","owner":"XXXXXXXXXXXX","logGroup":"/aws/rds/cluster/test-aurora-cluster/audit","logStream":"test-aurora-master-instance.audit.log.2.2020-05-08-10-22.0.3","subscriptionFilters":["Destination"],"logEvents":[{"id":"35434412929800762120104589099274971885390409568918831104","timestamp":1588934029529,"message":"1588934029529190,test-aurora-master-instance,rdsadmin,localhost,3,3789,QUERY,mysql,'select @@session.tx_read_only',0"}]}{"messageType":"DATA_MESSAGE","owner":"XXXXXXXXXXXX","logGroup":"/aws/rds/cluster/test-aurora-cluster/audit","logStream":"test-aurora-master-instance.audit.log.0.2020-05-08-10-22.0.3","subscriptionFilters":["Destination"],"logEvents":[{"id":"35434412927548386855052996162013805543482168054050848768","timestamp":1588934029428,"message":"1588934029428140,test-aurora-master-instance,rdsadmin,localhost,3,3786,QUERY,mysql,'SET @@sql_log_bin=off',0"},{"id":"35434412928551920388986874203382912865751344321819967489","timestamp":1588934029473,"message":"1588934029473859,test-aurora-master-instance,rdsadmin,localhost,3,3788,WRITE,mysql,rds_heartbeat2,"}]}{"messageType":"DATA_MESSAGE","owner":"XXXXXXXXXXXX","logGroup":"/aws/rds/cluster/test-aurora-cluster/audit","logStream":"test-aurora-master-instance.audit.log.1.2020-05-08-10-22.0.3","subscriptionFilters":["Destination"],"logEvents":[{"id":"35434412928440416662994221087716244297993433824790904832","timestamp":1588934029468,"message":"1588934029468081,test-aurora-master-instance,rdsadmin,localhost,3,3787,QUERY,mysql,'select @@session.tx_read_only',0"},{"id":"35434412929510852432523690998509958775080555177077964801","timestamp":1588934029516,"message":"1588934029516681,test-aurora-master-instance,rdsadmin,localhost,3,3788,QUERY,mysql,'INSERT INTO mysql.rds_heartbeat2(id, value) values (1,1588934029428) ON DUPLICATE KEY UPDATE value = 1588934029428',0"},{"id":"35434412930670491182847283401869816125258269975388946434","timestamp":1588934029568,"message":"1588934029568797,test-aurora-master-instance,rdsadmin,localhost,3,3790,QUERY,mysql,'COMMIT',0"}]}{"messageType":"DATA_MESSAGE","owner":"XXXXXXXXXXXX","logGroup":"/aws/rds/cluster/test-aurora-cluster/audit","logStream":"test-aurora-master-instance.audit.log.1.2020-05-08-10-22.0.3","subscriptionFilters":["Destination"],"logEvents":[{"id":"35434412971859967564533344346683415762524399170967961600","timestamp":1588934031415,"message":"1588934031415444,test-aurora-master-instance,rdsadmin,localhost,3,3795,QUERY,mysql,'SET @@sql_log_bin=off',0"}]}{"messageType":"DATA_MESSAGE","owner":"XXXXXXXXXXXX","logGroup":"/aws/rds/cluster/test-aurora-cluster/audit","logStream":"test-aurora-master-instance.audit.log.0.2020-05-08-10-22.0.0","subscriptionFilters":["Destination"],"logEvents":[{"id":"35434412973465621218827549212830711225717307494644121600","timestamp":1588934031487,"message":"1588934031487938,test-aurora-master-instance,rdsadmin,localhost,3,3796,QUERY,mysql,'select @@session.tx_read_only',0"},{"id":"35434412973487921964026079835972246943989955856150102017","timestamp":1588934031488,"message":"1588934031488740,test-aurora-master-instance,rdsadmin,localhost,3,3797,WRITE,mysql,rds_heartbeat2,"},{"id":"35434412974469154752761427254199818547986483762413240322","timestamp":1588934031532,"message":"1588934031532851,test-aurora-master-instance,rdsadmin,localhost,3,3798,QUERY,mysql,'select @@session.tx_read_only',0"},{"id":"35434412975717996483879142150125818771254792006748143619","timestamp":1588934031588,"message":"1588934031588794,test-aurora-master-instance,rdsadmin,localhost,3,3799,QUERY,mysql,'COMMIT',0"},{"id":"35434412985820234058813514433241499148764499768957272068","timestamp":1588934032041,"message":"1588934032041466,test-aurora-master-instance,admin,10.0.1.54,40,3801,QUERY,,'select now()',0"},{"id":"35434412985842534804012045056383034867037148130463252485","timestamp":1588934032042,"message":"1588934032042572,test-aurora-master-instance,admin,10.0.1.54,40,3802,QUERY,,'select user()',0"},{"id":"35434412985887136294409106302666106303582444853475213318","timestamp":1588934032044,"message":"1588934032044129,test-aurora-master-instance,rdsadmin,localhost,2,3804,QUERY,mysql,'set local oscar_local_only_replica_host_status=1',0"},{"id":"35434412985887136294409106302666106303582444853475213319","timestamp":1588934032044,"message":"1588934032044153,test-aurora-master-instance,admin,10.0.1.54,40,3803,READ,mysql,user,"},{"id":"35434412985887136294409106302666106303582444853475213320","timestamp":1588934032044,"message":"1588934032044264,test-aurora-master-instance,admin,10.0.1.54,40,3803,QUERY,mysql,'select host,user,authentication_string from mysql.user',0"},{"id":"35434412985887136294409106302666106303582444853475213321","timestamp":1588934032044,"message":"1588934032044753,test-aurora-master-instance,rdsadmin,localhost,2,3805,QUERY,mysql,'SELECT durable_lsn, current_read_point, server_id, last_update_timestamp FROM information_schema.replica_host_status',0"},{"id":"35434412985909437039607636925807642021855093214981193738","timestamp":1588934032045,"message":"1588934032045354,test-aurora-master-instance,rdsadmin,localhost,2,3806,QUERY,mysql,'set local oscar_local_only_replica_host_status=0',0"}]}{"messageType":"DATA_MESSAGE","owner":"XXXXXXXXXXXX","logGroup":"/aws/rds/cluster/test-aurora-cluster/audit","logStream":"test-aurora-master-instance.audit.log.3.2020-05-08-10-22.0.2","subscriptionFilters":["Destination"],"logEvents":[{"id":"35434412973710929416011386067329825380935958304238665728","timestamp":1588934031498,"message":"1588934031498452,test-aurora-master-instance,rdsadmin,localhost,3,3797,QUERY,mysql,'INSERT INTO mysql.rds_heartbeat2(id, value) values (1,1588934031415) ON DUPLICATE KEY UPDATE value = 1588934031415',0"},{"id":"35434412976164011387849754612898754390927278069896511489","timestamp":1588934031608,"message":"1588934031608371,test-aurora-master-instance,rdsadmin,localhost,3,3800,QUERY,mysql,'SET @@sql_log_bin=on',0"}]}{"messageType":"DATA_MESSAGE","owner":"XXXXXXXXXXXX","logGroup":"/aws/rds/cluster/test-aurora-cluster/audit","logStream":"test-aurora-master-instance.audit.log.2.2020-05-08-10-22.0.2","subscriptionFilters":["Destination"],"logEvents":[{"id":"35434413002478890722115889921126861524238751421182902272","timestamp":1588934032788,"message":"1588934032788148,test-aurora-master-instance,admin,10.0.1.54,40,3807,QUERY,mysql,'select now()',0"}]}{"messageType":"DATA_MESSAGE","owner":"XXXXXXXXXXXX","logGroup":"/aws/rds/cluster/test-aurora-cluster/audit","logStream":"test-aurora-master-instance.audit.log.3.2020-05-08-10-22.0.2","subscriptionFilters":["Destination"],"logEvents":[{"id":"35434413018535427265057938584227977259756395433104179200","timestamp":1588934033508,"message":"1588934033508241,test-aurora-master-instance,rdsadmin,localhost,3,3810,QUERY,mysql,'INSERT INTO mysql.rds_heartbeat2(id, value) values (1,1588934033436) ON DUPLICATE KEY UPDATE value = 1588934033436',0"}]}{"messageType":"DATA_MESSAGE","owner":"XXXXXXXXXXXX","logGroup":"/aws/rds/cluster/test-aurora-cluster/audit","logStream":"test-aurora-master-instance.audit.log.0.2020-05-08-10-22.0.1","subscriptionFilters":["Destination"],"logEvents":[{"id":"35434413016907472865565203094980709992037281047195680768","timestamp":1588934033435,"message":"1588934033435781,test-aurora-master-instance,rdsadmin,localhost,3,3808,QUERY,mysql,'SET @@sql_log_bin=off',0"},{"id":"35434413019538960798991816625681924748209787704901369857","timestamp":1588934033553,"message":"1588934033553002,test-aurora-master-instance,rdsadmin,localhost,3,3812,QUERY,mysql,'COMMIT',0"},{"id":"35434413020430990606933041551343353479115722165140586498","timestamp":1588934033593,"message":"1588934033593076,test-aurora-master-instance,rdsadmin,localhost,3,3813,QUERY,mysql,'SET @@sql_log_bin=on',0"}]}{"messageType":"DATA_MESSAGE","owner":"XXXXXXXXXXXX","logGroup":"/aws/rds/cluster/test-aurora-cluster/audit","logStream":"test-aurora-master-instance.audit.log.1.2020-05-08-10-22.0.1","subscriptionFilters":["Destination"],"logEvents":[{"id":"35434413018803036207440306062047480134402972585893101568","timestamp":1588934033520,"message":"1588934033520220,test-aurora-master-instance,rdsadmin,localhost,3,3811,QUERY,mysql,'select @@session.tx_read_only',0"}]}{"messageType":"DATA_MESSAGE","owner":"XXXXXXXXXXXX","logGroup":"/aws/rds/cluster/test-aurora-cluster/audit","logStream":"test-aurora-master-instance.audit.log.2.2020-05-08-10-22.0.3","subscriptionFilters":["Destination"],"logEvents":[{"id":"35434413018156314596682917990868198141171612945193107456","timestamp":1588934033491,"message":"1588934033491612,test-aurora-master-instance,rdsadmin,localhost,3,3809,QUERY,mysql,'select @@session.tx_read_only',0"},{"id":"35434413018357021303469693599142019605625448198746931201","timestamp":1588934033500,"message":"1588934033500959,test-aurora-master-instance,rdsadmin,localhost,3,3810,WRITE,mysql,rds_heartbeat2,"},{"id":"35434413030711634143455658819552807528672640473060081666","timestamp":1588934034054,"message":"1588934034054144,test-aurora-master-instance,rdsadmin,localhost,2,3814,QUERY,mysql,'set local oscar_local_only_replica_host_status=1',0"},{"id":"35434413030756235633852720065835878965217937196072042499","timestamp":1588934034056,"message":"1588934034056258,test-aurora-master-instance,rdsadmin,localhost,2,3815,QUERY,mysql,'SELECT durable_lsn, current_read_point, server_id, last_update_timestamp FROM information_schema.replica_host_status',0"},{"id":"35434413030756235633852720065835878965217937196072042500","timestamp":1588934034056,"message":"1588934034056344,test-aurora-master-instance,rdsadmin,localhost,2,3816,QUERY,mysql,'set local oscar_local_only_replica_host_status=0',0"}]}{"messageType":"DATA_MESSAGE","owner":"XXXXXXXXXXXX","logGroup":"/aws/rds/cluster/test-aurora-cluster/audit","logStream":"test-aurora-master-instance.audit.log.0.2020-05-08-10-22.0.0","subscriptionFilters":["Destination"],"logEvents":[{"id":"35434413062155684873383837451596295216840663405941882880","timestamp":1588934035464,"message":"1588934035464006,test-aurora-master-instance,rdsadmin,localhost,3,3817,QUERY,mysql,'SET @@sql_log_bin=off',0"},{"id":"35434413065166285475185471575703617183648192209249239041","timestamp":1588934035599,"message":"1588934035599881,test-aurora-master-instance,rdsadmin,localhost,3,3822,QUERY,mysql,'SET @@sql_log_bin=on',0"}]}{"messageType":"DATA_MESSAGE","owner":"XXXXXXXXXXXX","logGroup":"/aws/rds/cluster/test-aurora-cluster/audit","logStream":"test-aurora-master-instance.audit.log.2.2020-05-08-10-22.0.2","subscriptionFilters":["Destination"],"logEvents":[{"id":"35434413063226120642913307362345302541264930719570460672","timestamp":1588934035512,"message":"1588934035512954,test-aurora-master-instance,rdsadmin,localhost,3,3819,QUERY,mysql,'INSERT INTO mysql.rds_heartbeat2(id, value) values (1,1588934035464) ON DUPLICATE KEY UPDATE value = 1588934035464',0"},{"id":"35434413064408060138435430388846695609715293879387422721","timestamp":1588934035565,"message":"1588934035565223,test-aurora-master-instance,rdsadmin,localhost,3,3820,QUERY,mysql,'select @@session.tx_read_only',0"}]}{"messageType":"DATA_MESSAGE","owner":"XXXXXXXXXXXX","logGroup":"/aws/rds/cluster/test-aurora-cluster/audit","logStream":"test-aurora-master-instance.audit.log.3.2020-05-08-10-22.0.3","subscriptionFilters":["Destination"],"logEvents":[{"id":"35434413075803740934884578814122100732163079013086265344","timestamp":1588934036076,"message":"1588934036076439,test-aurora-master-instance,rdsadmin,localhost,2,3823,QUERY,mysql,'set local oscar_local_only_replica_host_status=1',0"},{"id":"35434413075803740934884578814122100732163079013086265345","timestamp":1588934036076,"message":"1588934036076948,test-aurora-master-instance,rdsadmin,localhost,2,3824,QUERY,mysql,'SELECT durable_lsn, current_read_point, server_id, last_update_timestamp FROM information_schema.replica_host_status',0"}]}{"messageType":"DATA_MESSAGE","owner":"XXXXXXXXXXXX","logGroup":"/aws/rds/cluster/test-aurora-cluster/audit","logStream":"test-aurora-master-instance.audit.log.1.2020-05-08-10-22.0.0","subscriptionFilters":["Destination"],"logEvents":[{"id":"35434413062534797541758858045034960841051843236826775552","timestamp":1588934035481,"message":"1588934035481668,test-aurora-master-instance,rdsadmin,localhost,3,3818,QUERY,mysql,'select @@session.tx_read_only',0"},{"id":"35434413062690902758148572407025710868960381767368638465","timestamp":1588934035488,"message":"1588934035488199,test-aurora-master-instance,rdsadmin,localhost,3,3819,WRITE,mysql,rds_heartbeat2,"},{"id":"35434413064787172806810450982330068386589327748930797570","timestamp":1588934035582,"message":"1588934035582626,test-aurora-master-instance,rdsadmin,localhost,3,3821,QUERY,mysql,'COMMIT',0"},{"id":"35434413075826041680083109437390248931550266694391103491","timestamp":1588934036077,"message":"1588934036077716,test-aurora-master-instance,rdsadmin,localhost,2,3825,QUERY,mysql,'set local oscar_local_only_replica_host_status=0',0"}]}{"messageType":"DATA_MESSAGE","owner":"XXXXXXXXXXXX","logGroup":"/aws/rds/cluster/test-aurora-cluster/audit","logStream":"test-aurora-master-instance.audit.log.0.2020-05-08-10-22.0.3","subscriptionFilters":["Destination"],"logEvents":[{"id":"35434413105508333539327368841171619188659342205864968192","timestamp":1588934037408,"message":"1588934037408567,test-aurora-master-instance,rdsadmin,localhost,3,3827,QUERY,mysql,'select @@session.tx_read_only',0"},{"id":"35434413119156389600828110203791478771520139447524982785","timestamp":1588934038020,"message":"1588934038020956,test-aurora-master-instance,rdsadmin,localhost,2,3832,QUERY,mysql,'set local oscar_local_only_replica_host_status=1',0"}]}{"messageType":"DATA_MESSAGE","owner":"XXXXXXXXXXXX","logGroup":"/aws/rds/cluster/test-aurora-cluster/audit","logStream":"test-aurora-master-instance.audit.log.1.2020-05-08-10-22.0.2","subscriptionFilters":["Destination"],"logEvents":[{"id":"35434413106355761856871532520452274254646652998912311296","timestamp":1588934037446,"message":"1588934037446243,test-aurora-master-instance,rdsadmin,localhost,3,3828,WRITE,mysql,rds_heartbeat2,"}]}{"messageType":"DATA_MESSAGE","owner":"XXXXXXXXXXXX","logGroup":"/aws/rds/cluster/test-aurora-cluster/audit","logStream":"test-aurora-master-instance.audit.log.2.2020-05-08-10-22.0.1","subscriptionFilters":["Destination"],"logEvents":[{"id":"35434413105441431303731776971687616144472375342442610688","timestamp":1588934037405,"message":"1588934037405151,test-aurora-master-instance,rdsadmin,localhost,3,3826,QUERY,mysql,'SET @@sql_log_bin=off',0"},{"id":"35434413106623370799253899998189009212922738502259572737","timestamp":1588934037458,"message":"1588934037458530,test-aurora-master-instance,rdsadmin,localhost,3,3828,QUERY,mysql,'INSERT INTO mysql.rds_heartbeat2(id, value) values (1,1588934037405) ON DUPLICATE KEY UPDATE value = 1588934037405',0"},{"id":"35434413107493099861996594300708902225556024600992808962","timestamp":1588934037497,"message":"1588934037497986,test-aurora-master-instance,rdsadmin,localhost,3,3829,QUERY,mysql,'select @@session.tx_read_only',0"},{"id":"35434413107760708804378961778407330844827804939064573955","timestamp":1588934037509,"message":"1588934037509476,test-aurora-master-instance,rdsadmin,localhost,3,3830,QUERY,mysql,'COMMIT',0"},{"id":"35434413107961415511165737386681152309281640192618397700","timestamp":1588934037518,"message":"1588934037518034,test-aurora-master-instance,rdsadmin,localhost,3,3831,QUERY,mysql,'SET @@sql_log_bin=on',0"},{"id":"35434413119178690346026640826873618600423766030126546949","timestamp":1588934038021,"message":"1588934038021504,test-aurora-master-instance,rdsadmin,localhost,2,3833,QUERY,mysql,'SELECT durable_lsn, current_read_point, server_id, last_update_timestamp FROM information_schema.replica_host_status',0"},{"id":"35434413119178690346026640826873618600423766030126546950","timestamp":1588934038021,"message":"1588934038021987,test-aurora-master-instance,rdsadmin,localhost,2,3834,QUERY,mysql,'set local oscar_local_only_replica_host_status=0',0"}]}{"messageType":"DATA_MESSAGE","owner":"XXXXXXXXXXXX","logGroup":"/aws/rds/cluster/test-aurora-cluster/audit","logStream":"test-aurora-master-instance.audit.log.3.2020-05-08-10-22.0.3","subscriptionFilters":["Destination"],"logEvents":[{"id":"35434413154213161052918249784591193988084389455363440640","timestamp":1588934039592,"message":"1588934039592502,test-aurora-master-instance,rdsadmin,localhost,3,3840,QUERY,mysql,'SET @@sql_log_bin=on',0"},{"id":"35434413165252029926190908239651374533045328400823746561","timestamp":1588934040087,"message":"1588934040087040,test-aurora-master-instance,rdsadmin,localhost,2,3847,QUERY,mysql,'set local oscar_local_only_replica_host_status=1',0"},{"id":"35434413165252029926190908239651374533045328400823746562","timestamp":1588934040087,"message":"1588934040087693,test-aurora-master-instance,rdsadmin,localhost,2,3848,QUERY,mysql,'SELECT durable_lsn, current_read_point, server_id, last_update_timestamp FROM information_schema.replica_host_status',0"},{"id":"35434413165252029926190908239651374533045328400823746563","timestamp":1588934040087,"message":"1588934040087855,test-aurora-master-instance,rdsadmin,localhost,2,3849,QUERY,mysql,'set local oscar_local_only_replica_host_status=0',0"}]}{"messageType":"DATA_MESSAGE","owner":"XXXXXXXXXXXX","logGroup":"/aws/rds/cluster/test-aurora-cluster/audit","logStream":"test-aurora-master-instance.audit.log.0.2020-05-08-10-22.0.2","subscriptionFilters":["Destination"],"logEvents":[{"id":"35434413153744845403749106698666207735361885022096457728","timestamp":1588934039571,"message":"1588934039571317,test-aurora-master-instance,rdsadmin,localhost,3,3839,QUERY,mysql,'COMMIT',0"},{"id":"35434413164895218003014418269434066871686065775086665729","timestamp":1588934040071,"message":"1588934040071684,test-aurora-master-instance,rdsadmin,localhost,39,3841,READ,mysql,rds_history,"},{"id":"35434413164895218003014418269434066871686065775086665730","timestamp":1588934040071,"message":"1588934040071809,test-aurora-master-instance,rdsadmin,localhost,39,3841,QUERY,mysql,'SELECT count(*) from mysql.rds_history WHERE action = \\'disable set master\\' ORDER BY action_timestamp LIMIT 1',0"},{"id":"35434413164895218003014418269434066871686065775086665731","timestamp":1588934040071,"message":"1588934040071881,test-aurora-master-instance,rdsadmin,localhost,2,3842,QUERY,mysql,'set local oscar_local_only_replica_host_status=1',0"},{"id":"35434413164917518748212948892575602589958714136592646148","timestamp":1588934040072,"message":"1588934040072759,test-aurora-master-instance,rdsadmin,localhost,2,3843,QUERY,mysql,'SELECT durable_lsn, current_read_point, server_id, last_update_timestamp FROM information_schema.replica_host_status',0"},{"id":"35434413164917518748212948892575602589958714136592646149","timestamp":1588934040072,"message":"1588934040072859,test-aurora-master-instance,rdsadmin,localhost,39,3845,QUERY,mysql,'SELECT @@aurora_version',0"},{"id":"35434413164939819493411479515717138308231362498098626566","timestamp":1588934040073,"message":"1588934040073219,test-aurora-master-instance,rdsadmin,localhost,2,3844,QUERY,mysql,'set local oscar_local_only_replica_host_status=0',0"},{"id":"35434413165006721729007071385141745463049307582616567815","timestamp":1588934040076,"message":"1588934040076099,test-aurora-master-instance,rdsadmin,localhost,39,3846,READ,mysql,rds_replication_status,"},{"id":"35434413165006721729007071385141745463049307582616567816","timestamp":1588934040076,"message":"1588934040076201,test-aurora-master-instance,rdsadmin,localhost,39,3846,QUERY,mysql,'SELECT count(*) from mysql.rds_replication_status WHERE master_host IS NOT NULL and master_port IS NOT NULL ORDER BY action_timestamp LIMIT 1',0"}]}{"messageType":"DATA_MESSAGE","owner":"XXXXXXXXXXXX","logGroup":"/aws/rds/cluster/test-aurora-cluster/audit","logStream":"test-aurora-master-instance.audit.log.2.2020-05-08-10-22.0.3","subscriptionFilters":["Destination"],"logEvents":[{"id":"35434413151224861196315146283716238736014173450477633536","timestamp":1588934039458,"message":"1588934039458710,test-aurora-master-instance,rdsadmin,localhost,3,3835,QUERY,mysql,'SET @@sql_log_bin=off',0"},{"id":"35434413151492470138697513761414667355285953788549398529","timestamp":1588934039470,"message":"1588934039470019,test-aurora-master-instance,rdsadmin,localhost,3,3836,QUERY,mysql,'select @@session.tx_read_only',0"},{"id":"35434413151559372374293105630839274510103898873067339778","timestamp":1588934039473,"message":"1588934039473756,test-aurora-master-instance,rdsadmin,localhost,3,3837,WRITE,mysql,rds_heartbeat2,"},{"id":"35434413151760079081079881239113095974557734126621163523","timestamp":1588934039482,"message":"1588934039482096,test-aurora-master-instance,rdsadmin,localhost,3,3837,QUERY,mysql,'INSERT INTO mysql.rds_heartbeat2(id, value) values (1,1588934039459) ON DUPLICATE KEY UPDATE value = 1588934039459',0"},{"id":"35434413153499537206565269844152881999824306324087635972","timestamp":1588934039560,"message":"1588934039560194,test-aurora-master-instance,rdsadmin,localhost,3,3838,QUERY,mysql,'select @@session.tx_read_only',0"}]}{"messageType":"DATA_MESSAGE","owner":"XXXXXXXXXXXX","logGroup":"/aws/rds/cluster/test-aurora-cluster/audit","logStream":"test-aurora-master-instance.audit.log.2.2020-05-08-10-22.0.1","subscriptionFilters":["Destination"],"logEvents":[{"id":"35434413196517674694530841886555395942832766583318577152","timestamp":1588934041489,"message":"1588934041489477,test-aurora-master-instance,rdsadmin,localhost,3,3853,QUERY,mysql,'select @@session.tx_read_only',0"}]}{"messageType":"DATA_MESSAGE","owner":"XXXXXXXXXXXX","logGroup":"/aws/rds/cluster/test-aurora-cluster/audit","logStream":"test-aurora-master-instance.audit.log.0.2020-05-08-10-22.0.1","subscriptionFilters":["Destination"],"logEvents":[{"id":"35434413195179629982619004498100987412231518924172296192","timestamp":1588934041429,"message":"1588934041429392,test-aurora-master-instance,rdsadmin,localhost,3,3851,QUERY,mysql,'select @@session.tx_read_only',0"},{"id":"35434413195224231473016065744384058848776815647184257025","timestamp":1588934041431,"message":"1588934041431603,test-aurora-master-instance,rdsadmin,localhost,3,3852,WRITE,mysql,rds_heartbeat2,"},{"id":"35434413195380336689405780106374808876685354177726119938","timestamp":1588934041438,"message":"1588934041438260,test-aurora-master-instance,rdsadmin,localhost,3,3852,QUERY,mysql,'INSERT INTO mysql.rds_heartbeat2(id, value) values (1,1588934041404) ON DUPLICATE KEY UPDATE value = 1588934041404',0"},{"id":"35434413197231298540883821827122273493315168182722494467","timestamp":1588934041521,"message":"1588934041521912,test-aurora-master-instance,rdsadmin,localhost,3,3854,QUERY,mysql,'COMMIT',0"},{"id":"35434413197432005247670597435396094957769003436276318212","timestamp":1588934041530,"message":"1588934041530631,test-aurora-master-instance,rdsadmin,localhost,3,3855,QUERY,mysql,'SET @@sql_log_bin=on',0"}]}{"messageType":"DATA_MESSAGE","owner":"XXXXXXXXXXXX","logGroup":"/aws/rds/cluster/test-aurora-cluster/audit","logStream":"test-aurora-master-instance.audit.log.1.2020-05-08-10-22.0.3","subscriptionFilters":["Destination"],"logEvents":[{"id":"35434413194622111352655738919481342712132884346400079872","timestamp":1588934041404,"message":"1588934041404417,test-aurora-master-instance,rdsadmin,localhost,3,3850,QUERY,mysql,'SET @@sql_log_bin=off',0"},{"id":"35434413210477941188811011973113238403985869377152155649","timestamp":1588934042115,"message":"1588934042115152,test-aurora-master-instance,rdsadmin,localhost,2,3856,QUERY,mysql,'set local oscar_local_only_replica_host_status=1',0"},{"id":"35434413210477941188811011973113238403985869377152155650","timestamp":1588934042115,"message":"1588934042115864,test-aurora-master-instance,rdsadmin,localhost,2,3857,QUERY,mysql,'SELECT durable_lsn, current_read_point, server_id, last_update_timestamp FROM information_schema.replica_host_status',0"},{"id":"35434413210500241934009542596254774122258517738658136067","timestamp":1588934042116,"message":"1588934042116521,test-aurora-master-instance,rdsadmin,localhost,2,3858,QUERY,mysql,'set local oscar_local_only_replica_host_status=0',0"}]}{"messageType":"DATA_MESSAGE","owner":"XXXXXXXXXXXX","logGroup":"/aws/rds/cluster/test-aurora-cluster/audit","logStream":"test-aurora-master-instance.audit.log.2.2020-05-08-10-22.0.0","subscriptionFilters":["Destination"],"logEvents":[{"id":"35434413240517044971231761347323395500033791842166308864","timestamp":1588934043462,"message":"1588934043462741,test-aurora-master-instance,admin,10.0.1.54,40,0,DISCONNECT,,,0"},{"id":"35434413241342172543577394403560217076121781217887584257","timestamp":1588934043499,"message":"1588934043499244,test-aurora-master-instance,rdsadmin,localhost,3,3863,QUERY,mysql,'select @@session.tx_read_only',0"},{"id":"35434413241409074779172986272984824230939726302405525506","timestamp":1588934043502,"message":"1588934043502943,test-aurora-master-instance,rdsadmin,localhost,3,3864,QUERY,mysql,'COMMIT',0"}]}{"messageType":"DATA_MESSAGE","owner":"XXXXXXXXXXXX","logGroup":"/aws/rds/cluster/test-aurora-cluster/audit","logStream":"test-aurora-master-instance.audit.log.1.2020-05-08-10-22.0.3","subscriptionFilters":["Destination"],"logEvents":[{"id":"35434413240762353168415598201777742261094812544761462784","timestamp":1588934043473,"message":"1588934043473855,test-aurora-master-instance,rdsadmin,localhost,3,3862,WRITE,mysql,rds_heartbeat2,"},{"id":"35434413240940759130003843186910028007275999436809306113","timestamp":1588934043481,"message":"1588934043481453,test-aurora-master-instance,rdsadmin,localhost,3,3862,QUERY,mysql,'INSERT INTO mysql.rds_heartbeat2(id, value) values (1,1588934043401) ON DUPLICATE KEY UPDATE value = 1588934043401',0"}]}{"messageType":"DATA_MESSAGE","owner":"XXXXXXXXXXXX","logGroup":"/aws/rds/cluster/test-aurora-cluster/audit","logStream":"test-aurora-master-instance.audit.log.0.2020-05-08-10-22.0.0","subscriptionFilters":["Destination"],"logEvents":[{"id":"35434413239112098023724332089356145553500786395296497664","timestamp":1588934043399,"message":"1588934043399844,test-aurora-master-instance,rdsadmin,localhost,3,3859,QUERY,mysql,'SET @@sql_log_bin=off',0"},{"id":"35434413240071030067261148884442181439224665940053655553","timestamp":1588934043442,"message":"1588934043442378,test-aurora-master-instance,rdsadmin,localhost,3,3860,QUERY,mysql,'select @@session.tx_read_only',0"},{"id":"35434413242323405332312741821737288984762150452157677570","timestamp":1588934043543,"message":"1588934043543158,test-aurora-master-instance,rdsadmin,localhost,3,3865,QUERY,mysql,'SET @@sql_log_bin=on',0"},{"id":"35434413253830589854754543362769719613448704989243572227","timestamp":1588934044059,"message":"1588934044059026,test-aurora-master-instance,rdsadmin,localhost,2,3866,QUERY,mysql,'set local oscar_local_only_replica_host_status=1',0"},{"id":"35434413253830589854754543362769719613448704989243572228","timestamp":1588934044059,"message":"1588934044059777,test-aurora-master-instance,rdsadmin,localhost,2,3867,QUERY,mysql,'SELECT durable_lsn, current_read_point, server_id, last_update_timestamp FROM information_schema.replica_host_status',0"},{"id":"35434413253852890599953073985911255331721353350749552645","timestamp":1588934044060,"message":"1588934044060130,test-aurora-master-instance,rdsadmin,localhost,2,3868,QUERY,mysql,'set local oscar_local_only_replica_host_status=0',0"}]}{"messageType":"DATA_MESSAGE","owner":"XXXXXXXXXXXX","logGroup":"/aws/rds/cluster/test-aurora-cluster/audit","logStream":"test-aurora-master-instance.audit.log.2.2020-05-08-10-22.0.0","subscriptionFilters":["Destination"],"logEvents":[{"id":"35434413287482414359337253685694540020518542072941117440","timestamp":1588934045568,"message":"1588934045568713,test-aurora-master-instance,rdsadmin,localhost,3,3874,QUERY,mysql,'SET @@sql_log_bin=on',0"}]}{"messageType":"DATA_MESSAGE","owner":"XXXXXXXXXXXX","logGroup":"/aws/rds/cluster/test-aurora-cluster/audit","logStream":"test-aurora-master-instance.audit.log.3.2020-05-08-10-22.0.3","subscriptionFilters":["Destination"],"logEvents":[{"id":"35434413285230039094285660748446563616566170325831909376","timestamp":1588934045467,"message":"1588934045467786,test-aurora-master-instance,rdsadmin,localhost,3,3870,QUERY,mysql,'select @@session.tx_read_only',0"},{"id":"35434413285252339839484191371588099334838818687337889793","timestamp":1588934045468,"message":"1588934045468317,test-aurora-master-instance,rdsadmin,localhost,3,3871,WRITE,mysql,rds_heartbeat2,"}]}{"messageType":"DATA_MESSAGE","owner":"XXXXXXXXXXXX","logGroup":"/aws/rds/cluster/test-aurora-cluster/audit","logStream":"test-aurora-master-instance.audit.log.1.2020-05-08-10-22.0.0","subscriptionFilters":["Destination"],"logEvents":[{"id":"35434413283958896617969415229295757745404118266350731264","timestamp":1588934045410,"message":"1588934045410099,test-aurora-master-instance,rdsadmin,localhost,3,3869,QUERY,mysql,'SET @@sql_log_bin=off',0"},{"id":"35434413285564550272263620095486329461034800294781321217","timestamp":1588934045482,"message":"1588934045482121,test-aurora-master-instance,rdsadmin,localhost,3,3871,QUERY,mysql,'INSERT INTO mysql.rds_heartbeat2(id, value) values (1,1588934045410) ON DUPLICATE KEY UPDATE value = 1588934045410',0"},{"id":"35434413286144369647425416297166258136123657693936812034","timestamp":1588934045508,"message":"1588934045508695,test-aurora-master-instance,rdsadmin,localhost,3,3872,QUERY,mysql,'select @@session.tx_read_only',0"},{"id":"35434413286590384551396028759996972501576624924056420355","timestamp":1588934045528,"message":"1588934045528549,test-aurora-master-instance,rdsadmin,localhost,3,3873,QUERY,mysql,'COMMIT',0"},{"id":"35434413298275975035426075286161688876444366353190158340","timestamp":1588934046052,"message":"1588934046052286,test-aurora-master-instance,rdsadmin,localhost,2,3875,QUERY,mysql,'set local oscar_local_only_replica_host_status=1',0"},{"id":"35434413298275975035426075286161688876444366353190158341","timestamp":1588934046052,"message":"1588934046052730,test-aurora-master-instance,rdsadmin,localhost,2,3876,QUERY,mysql,'SELECT durable_lsn, current_read_point, server_id, last_update_timestamp FROM information_schema.replica_host_status',0"},{"id":"35434413298298275780624605909303224594717014714696138758","timestamp":1588934046053,"message":"1588934046053374,test-aurora-master-instance,rdsadmin,localhost,2,3877,QUERY,mysql,'set local oscar_local_only_replica_host_status=0',0"}]}{"messageType":"DATA_MESSAGE","owner":"XXXXXXXXXXXX","logGroup":"/aws/rds/cluster/test-aurora-cluster/audit","logStream":"test-aurora-master-instance.audit.log.0.2020-05-08-10-22.0.0","subscriptionFilters":["Destination"],"logEvents":[{"id":"35434413331169574203258744422480236743867552145444962304","timestamp":1588934047527,"message":"1588934047527035,test-aurora-master-instance,rdsadmin,localhost,3,3882,QUERY,mysql,'COMMIT',0"},{"id":"35434413342141540840935811008115810134010546006387326977","timestamp":1588934048019,"message":"1588934048019039,test-aurora-master-instance,rdsadmin,localhost,2,3885,QUERY,mysql,'SELECT durable_lsn, current_read_point, server_id, last_update_timestamp FROM information_schema.replica_host_status',0"},{"id":"35434413342141540840935811008115810134010546006387326978","timestamp":1588934048019,"message":"1588934048019902,test-aurora-master-instance,rdsadmin,localhost,2,3886,QUERY,mysql,'set local oscar_local_only_replica_host_status=0',0"}]}{"messageType":"DATA_MESSAGE","owner":"XXXXXXXXXXXX","logGroup":"/aws/rds/cluster/test-aurora-cluster/audit","logStream":"test-aurora-master-instance.audit.log.3.2020-05-08-10-22.0.1","subscriptionFilters":["Destination"],"logEvents":[{"id":"35434413328894898193008620861918895996666662979509026816","timestamp":1588934047425,"message":"1588934047425126,test-aurora-master-instance,rdsadmin,localhost,3,3878,QUERY,mysql,'SET @@sql_log_bin=off',0"}]}{"messageType":"DATA_MESSAGE","owner":"XXXXXXXXXXXX","logGroup":"/aws/rds/cluster/test-aurora-cluster/audit","logStream":"test-aurora-master-instance.audit.log.2.2020-05-08-10-22.0.3","subscriptionFilters":["Destination"],"logEvents":[{"id":"35434413342119240095737280384928244138649348521972662272","timestamp":1588934048018,"message":"1588934048018631,test-aurora-master-instance,rdsadmin,localhost,2,3884,QUERY,mysql,'set local oscar_local_only_replica_host_status=1',0"}]}{"messageType":"DATA_MESSAGE","owner":"XXXXXXXXXXXX","logGroup":"/aws/rds/cluster/test-aurora-cluster/audit","logStream":"test-aurora-master-instance.audit.log.1.2020-05-08-10-22.0.0","subscriptionFilters":["Destination"],"logEvents":[{"id":"35434413330099138433729274511723810422736946151396868096","timestamp":1588934047479,"message":"1588934047479837,test-aurora-master-instance,rdsadmin,localhost,3,3879,QUERY,mysql,'select @@session.tx_read_only',0"},{"id":"35434413330277544395317519496856096168918133043444711425","timestamp":1588934047487,"message":"1588934047487454,test-aurora-master-instance,rdsadmin,localhost,3,3880,WRITE,mysql,rds_heartbeat2,"},{"id":"35434413330366747376111641989422239042008726489468633090","timestamp":1588934047491,"message":"1588934047491821,test-aurora-master-instance,rdsadmin,localhost,3,3880,QUERY,mysql,'INSERT INTO mysql.rds_heartbeat2(id, value) values (1,1588934047425) ON DUPLICATE KEY UPDATE value = 1588934047425',0"},{"id":"35434413330589754828096948220837596224735210104528437251","timestamp":1588934047501,"message":"1588934047501105,test-aurora-master-instance,rdsadmin,localhost,3,3881,QUERY,mysql,'select @@session.tx_read_only',0"},{"id":"35434413332396115189177928695301989404819727386512850948","timestamp":1588934047582,"message":"1588934047582362,test-aurora-master-instance,rdsadmin,localhost,3,3883,QUERY,mysql,'SET @@sql_log_bin=on',0"}]}{"messageType":"DATA_MESSAGE","owner":"XXXXXXXXXXXX","logGroup":"/aws/rds/cluster/test-aurora-cluster/audit","logStream":"test-aurora-master-instance.audit.log.0.2020-05-08-10-22.0.1","subscriptionFilters":["Destination"],"logEvents":[{"id":"35434413374722929575989051420225661305153121583425519616","timestamp":1588934049480,"message":"1588934049480313,test-aurora-master-instance,rdsadmin,localhost,3,3889,WRITE,mysql,rds_heartbeat2,"}]}{"messageType":"DATA_MESSAGE","owner":"XXXXXXXXXXXX","logGroup":"/aws/rds/cluster/test-aurora-cluster/audit","logStream":"test-aurora-master-instance.audit.log.1.2020-05-08-10-22.0.3","subscriptionFilters":["Destination"],"logEvents":[{"id":"35434413373830899768047826494527705063855584778123476992","timestamp":1588934049440,"message":"1588934049440572,test-aurora-master-instance,rdsadmin,localhost,3,3887,QUERY,mysql,'SET @@sql_log_bin=off',0"},{"id":"35434413374254613926819908334216883711035903646737104897","timestamp":1588934049459,"message":"1588934049459166,test-aurora-master-instance,rdsadmin,localhost,3,3888,QUERY,mysql,'select @@session.tx_read_only',0"},{"id":"35434413374945937027974357651604490977488002853422497794","timestamp":1588934049490,"message":"1588934049490147,test-aurora-master-instance,rdsadmin,localhost,3,3889,QUERY,mysql,'INSERT INTO mysql.rds_heartbeat2(id, value) values (1,1588934049440) ON DUPLICATE KEY UPDATE value = 1588934049440',0"},{"id":"35434413375235846715555255752444455315032431553000243203","timestamp":1588934049503,"message":"1588934049503193,test-aurora-master-instance,rdsadmin,localhost,3,3890,QUERY,mysql,'select @@session.tx_read_only',0"},{"id":"35434413376484688446672970648370455538300739797335146500","timestamp":1588934049559,"message":"1588934049559306,test-aurora-master-instance,rdsadmin,localhost,3,3891,QUERY,mysql,'COMMIT',0"},{"id":"35434413377376718254614195574031884269206674257574363141","timestamp":1588934049599,"message":"1588934049599849,test-aurora-master-instance,rdsadmin,localhost,3,3892,QUERY,mysql,'SET @@sql_log_bin=on',0"},{"id":"35434413387233647632364731002590671745717250043217707014","timestamp":1588934050041,"message":"1588934050041739,test-aurora-master-instance,rdsadmin,localhost,2,3893,QUERY,mysql,'set local oscar_local_only_replica_host_status=1',0"},{"id":"35434413387255948377563261625732207463989898404723687431","timestamp":1588934050042,"message":"1588934050042344,test-aurora-master-instance,rdsadmin,localhost,2,3894,QUERY,mysql,'SELECT durable_lsn, current_read_point, server_id, last_update_timestamp FROM information_schema.replica_host_status',0"},{"id":"35434413387255948377563261625732207463989898404723687432","timestamp":1588934050042,"message":"1588934050042712,test-aurora-master-instance,rdsadmin,localhost,2,3895,QUERY,mysql,'set local oscar_local_only_replica_host_status=0',0"}]}{"messageType":"DATA_MESSAGE","owner":"XXXXXXXXXXXX","logGroup":"/aws/rds/cluster/test-aurora-cluster/audit","logStream":"test-aurora-master-instance.audit.log.1.2020-05-08-10-22.0.3","subscriptionFilters":["Destination"],"logEvents":[{"id":"35434413417629563337961970346913077133920416149139423232","timestamp":1588934051404,"message":"1588934051404628,test-aurora-master-instance,rdsadmin,localhost,3,3896,QUERY,mysql,'SET @@sql_log_bin=off',0"},{"id":"35434413418766901343087032127131398765825482585944424449","timestamp":1588934051455,"message":"1588934051455511,test-aurora-master-instance,rdsadmin,localhost,3,3897,QUERY,mysql,'select @@session.tx_read_only',0"},{"id":"35434413418923006559476746489122148793734021116486287362","timestamp":1588934051462,"message":"1588934051462760,test-aurora-master-instance,rdsadmin,localhost,3,3898,WRITE,mysql,rds_heartbeat2,"},{"id":"35434413419101412521064991474254434539915208008534130691","timestamp":1588934051470,"message":"1588934051470681,test-aurora-master-instance,rdsadmin,localhost,3,3898,QUERY,mysql,'INSERT INTO mysql.rds_heartbeat2(id, value) values (1,1588934051404) ON DUPLICATE KEY UPDATE value = 1588934051404',0"},{"id":"35434413419993442329006216399915863270821142468773347332","timestamp":1588934051510,"message":"1588934051510196,test-aurora-master-instance,rdsadmin,localhost,3,3899,QUERY,mysql,'select @@session.tx_read_only',0"},{"id":"35434413420283352016587114500755827608365571168351092741","timestamp":1588934051523,"message":"1588934051523639,test-aurora-master-instance,rdsadmin,localhost,3,3900,QUERY,mysql,'COMMIT',0"},{"id":"35434413421019276608138625064426506311362967098048446470","timestamp":1588934051556,"message":"1588934051556177,test-aurora-master-instance,rdsadmin,localhost,3,3901,QUERY,mysql,'SET @@sql_log_bin=on',0"},{"id":"35434413432392656659389242866609722630413631466098458631","timestamp":1588934052066,"message":"1588934052066741,test-aurora-master-instance,rdsadmin,localhost,2,3902,QUERY,mysql,'set local oscar_local_only_replica_host_status=1',0"},{"id":"35434413432414957404587773489751258348686279827604439048","timestamp":1588934052067,"message":"1588934052067733,test-aurora-master-instance,rdsadmin,localhost,2,3903,QUERY,mysql,'SELECT durable_lsn, current_read_point, server_id, last_update_timestamp FROM information_schema.replica_host_status',0"},{"id":"35434413432437258149786304112892794066958928189110419465","timestamp":1588934052068,"message":"1588934052068161,test-aurora-master-instance,rdsadmin,localhost,2,3904,QUERY,mysql,'set local oscar_local_only_replica_host_status=0',0"}]}{"messageType":"DATA_MESSAGE","owner":"XXXXXXXXXXXX","logGroup":"/aws/rds/cluster/test-aurora-cluster/audit","logStream":"test-aurora-master-instance.audit.log.1.2020-05-08-10-22.0.0","subscriptionFilters":["Destination"],"logEvents":[{"id":"35434413462520963422604114733252262139940781143284973568","timestamp":1588934053417,"message":"1588934053417811,test-aurora-master-instance,rdsadmin,localhost,3,3905,QUERY,mysql,'SET @@sql_log_bin=off',0"},{"id":"35434413462699369384192359718384547886121968035332816897","timestamp":1588934053425,"message":"1588934053425997,test-aurora-master-instance,rdsadmin,localhost,3,3906,QUERY,mysql,'select @@session.tx_read_only',0"},{"id":"35434413463925910370111543991169012391117627918161739778","timestamp":1588934053480,"message":"1588934053480426,test-aurora-master-instance,rdsadmin,localhost,3,3907,WRITE,mysql,rds_heartbeat2,"},{"id":"35434413464015113350905666483735155264208221364185661443","timestamp":1588934053484,"message":"1588934053484124,test-aurora-master-instance,rdsadmin,localhost,3,3907,QUERY,mysql,'INSERT INTO mysql.rds_heartbeat2(id, value) values (1,1588934053418) ON DUPLICATE KEY UPDATE value = 1588934053418',0"},{"id":"35434413465152451356030728263953476896113287800990662660","timestamp":1588934053535,"message":"1588934053535903,test-aurora-master-instance,rdsadmin,localhost,3,3908,QUERY,mysql,'select @@session.tx_read_only',0"},{"id":"35434413465308556572420442625944226924021826331532525573","timestamp":1588934053542,"message":"1588934053542049,test-aurora-master-instance,rdsadmin,localhost,3,3909,QUERY,mysql,'COMMIT',0"},{"id":"35434413466401293087148443159879477119381596045325565958","timestamp":1588934053591,"message":"1588934053591255,test-aurora-master-instance,rdsadmin,localhost,3,3910,QUERY,mysql,'SET @@sql_log_bin=on',0"}]}{"messageType":"DATA_MESSAGE","owner":"XXXXXXXXXXXX","logGroup":"/aws/rds/cluster/test-aurora-cluster/audit","logStream":"test-aurora-master-instance.audit.log.0.2020-05-08-10-22.0.0","subscriptionFilters":["Destination"],"logEvents":[{"id":"35434413476637335133273999181911823447402450768413130752","timestamp":1588934054050,"message":"1588934054050833,test-aurora-master-instance,rdsadmin,localhost,2,3912,QUERY,mysql,'SELECT durable_lsn, current_read_point, server_id, last_update_timestamp FROM information_schema.replica_host_status',0"},{"id":"35434413476659635878472529805053359165675099129919111169","timestamp":1588934054051,"message":"1588934054051241,test-aurora-master-instance,rdsadmin,localhost,2,3913,QUERY,mysql,'set local oscar_local_only_replica_host_status=0',0"}]}{"messageType":"DATA_MESSAGE","owner":"XXXXXXXXXXXX","logGroup":"/aws/rds/cluster/test-aurora-cluster/audit","logStream":"test-aurora-master-instance.audit.log.2.2020-05-08-10-22.0.1","subscriptionFilters":["Destination"],"logEvents":[{"id":"35434413476637335133273999181880766054771507251582730240","timestamp":1588934054050,"message":"1588934054050385,test-aurora-master-instance,rdsadmin,localhost,2,3911,QUERY,mysql,'set local oscar_local_only_replica_host_status=1',0"}]}{"messageType":"DATA_MESSAGE","owner":"XXXXXXXXXXXX","logGroup":"/aws/rds/cluster/test-aurora-cluster/audit","logStream":"test-aurora-master-instance.audit.log.1.2020-05-08-10-22.0.2","subscriptionFilters":["Destination"],"logEvents":[{"id":"35434413498603569153826662977451625230672132515161899008","timestamp":1588934055035,"message":"1588934055035306,test-aurora-master-instance,rdsadmin,localhost,39,3914,QUERY,mysql,'SELECT @@aurora_version',0"},{"id":"35434413500075418336929684104792982636666924374556606465","timestamp":1588934055101,"message":"1588934055101343,test-aurora-master-instance,rdsadmin,localhost,2,3915,QUERY,mysql,'set local oscar_local_only_replica_host_status=1',0"},{"id":"35434413500075418336929684104792982636666924374556606466","timestamp":1588934055101,"message":"1588934055101787,test-aurora-master-instance,rdsadmin,localhost,2,3916,QUERY,mysql,'SELECT durable_lsn, current_read_point, server_id, last_update_timestamp FROM information_schema.replica_host_status',0"},{"id":"35434413500097719082128214727934518354939572736062586883","timestamp":1588934055102,"message":"1588934055102324,test-aurora-master-instance,rdsadmin,localhost,2,3917,QUERY,mysql,'set local oscar_local_only_replica_host_status=0',0"}]}{"messageType":"DATA_MESSAGE","owner":"XXXXXXXXXXXX","logGroup":"/aws/rds/cluster/test-aurora-cluster/audit","logStream":"test-aurora-master-instance.audit.log.0.2020-05-08-10-22.0.1","subscriptionFilters":["Destination"],"logEvents":[{"id":"35434413511738708075761200009108333551378419001085526016","timestamp":1588934055624,"message":"1588934055624095,test-aurora-master-instance,rdsadmin,localhost,3,3923,QUERY,mysql,'SET @@sql_log_bin=on',0"}]}{"messageType":"DATA_MESSAGE","owner":"XXXXXXXXXXXX","logGroup":"/aws/rds/cluster/test-aurora-cluster/audit","logStream":"test-aurora-master-instance.audit.log.2.2020-05-08-10-22.0.3","subscriptionFilters":["Destination"],"logEvents":[{"id":"35434413508059085118003647190716008663265604560234283008","timestamp":1588934055459,"message":"1588934055459409,test-aurora-master-instance,rdsadmin,localhost,3,3918,QUERY,mysql,'SET @@sql_log_bin=off',0"},{"id":"35434413509174122377930178347792794576898022635533303809","timestamp":1588934055509,"message":"1588934055509527,test-aurora-master-instance,rdsadmin,localhost,3,3920,QUERY,mysql,'INSERT INTO mysql.rds_heartbeat2(id, value) values (1,1588934055459) ON DUPLICATE KEY UPDATE value = 1588934055459',0"},{"id":"35434413510333761128253770751152651927075737433844285442","timestamp":1588934055561,"message":"1588934055561237,test-aurora-master-instance,rdsadmin,localhost,3,3921,QUERY,mysql,'select @@session.tx_read_only',0"}]}{"messageType":"DATA_MESSAGE","owner":"XXXXXXXXXXXX","logGroup":"/aws/rds/cluster/test-aurora-cluster/audit","logStream":"test-aurora-master-instance.audit.log.1.2020-05-08-10-22.0.3","subscriptionFilters":["Destination"],"logEvents":[{"id":"35434413508861911945150749623776443830241067379373309952","timestamp":1588934055495,"message":"1588934055495279,test-aurora-master-instance,rdsadmin,localhost,3,3919,QUERY,mysql,'select @@session.tx_read_only',0"},{"id":"35434413508928814180746341493201050985059012463891251201","timestamp":1588934055498,"message":"1588934055498875,test-aurora-master-instance,rdsadmin,localhost,3,3920,WRITE,mysql,rds_heartbeat2,"},{"id":"35434413511114287210202342561071551375778551891477331970","timestamp":1588934055596,"message":"1588934055596101,test-aurora-master-instance,rdsadmin,localhost,3,3922,QUERY,mysql,'COMMIT',0"},{"id":"35434413521038118823548469859054946007107072761638617091","timestamp":1588934056041,"message":"1588934056041878,test-aurora-master-instance,rdsadmin,localhost,2,3924,QUERY,mysql,'set local oscar_local_only_replica_host_status=1',0"},{"id":"35434413521060419568747000482196481725379721123144597508","timestamp":1588934056042,"message":"1588934056042307,test-aurora-master-instance,rdsadmin,localhost,2,3925,QUERY,mysql,'SELECT durable_lsn, current_read_point, server_id, last_update_timestamp FROM information_schema.replica_host_status',0"},{"id":"35434413521060419568747000482196481725379721123144597509","timestamp":1588934056042,"message":"1588934056042659,test-aurora-master-instance,rdsadmin,localhost,2,3926,QUERY,mysql,'set local oscar_local_only_replica_host_status=0',0"}]}{"messageType":"DATA_MESSAGE","owner":"XXXXXXXXXXXX","logGroup":"/aws/rds/cluster/test-aurora-cluster/audit","logStream":"test-aurora-master-instance.audit.log.3.2020-05-08-10-22.0.1","subscriptionFilters":["Destination"],"logEvents":[{"id":"35434413556697010395998936264865502270781569178118717440","timestamp":1588934057640,"message":"1588934057640368,test-aurora-master-instance,rdsadmin,localhost,3,3932,QUERY,mysql,'SET @@sql_log_bin=on',0"}]}{"messageType":"DATA_MESSAGE","owner":"XXXXXXXXXXXX","logGroup":"/aws/rds/cluster/test-aurora-cluster/audit","logStream":"test-aurora-master-instance.audit.log.1.2020-05-08-10-22.0.1","subscriptionFilters":["Destination"],"logEvents":[{"id":"35434413552236861356292811636485876937708219865815121920","timestamp":1588934057440,"message":"1588934057440082,test-aurora-master-instance,rdsadmin,localhost,3,3927,QUERY,mysql,'SET @@sql_log_bin=off',0"},{"id":"35434413554489236621344404573780984483245704377919143937","timestamp":1588934057541,"message":"1588934057541468,test-aurora-master-instance,rdsadmin,localhost,3,3930,QUERY,mysql,'select @@session.tx_read_only',0"},{"id":"35434413565884917417793552999105736520569017107475136514","timestamp":1588934058052,"message":"1588934058052792,test-aurora-master-instance,rdsadmin,localhost,2,3933,QUERY,mysql,'set local oscar_local_only_replica_host_status=1',0"},{"id":"35434413565907218162992083622247272238841665468981116931","timestamp":1588934058053,"message":"1588934058053208,test-aurora-master-instance,rdsadmin,localhost,2,3934,QUERY,mysql,'SELECT durable_lsn, current_read_point, server_id, last_update_timestamp FROM information_schema.replica_host_status',0"},{"id":"35434413565907218162992083622247272238841665468981116932","timestamp":1588934058053,"message":"1588934058053979,test-aurora-master-instance,rdsadmin,localhost,2,3935,QUERY,mysql,'set local oscar_local_only_replica_host_status=0',0"}]}{"messageType":"DATA_MESSAGE","owner":"XXXXXXXXXXXX","logGroup":"/aws/rds/cluster/test-aurora-cluster/audit","logStream":"test-aurora-master-instance.audit.log.2.2020-05-08-10-22.0.2","subscriptionFilters":["Destination"],"logEvents":[{"id":"35434413552928184457447260953908799726113192428529975296","timestamp":1588934057471,"message":"1588934057471917,test-aurora-master-instance,rdsadmin,localhost,3,3928,QUERY,mysql,'select @@session.tx_read_only',0"},{"id":"35434413554043221717373792110985585639745610503828996097","timestamp":1588934057521,"message":"1588934057521429,test-aurora-master-instance,rdsadmin,localhost,3,3929,WRITE,mysql,rds_heartbeat2,"},{"id":"35434413554288529914557628965542478540744742480394780674","timestamp":1588934057532,"message":"1588934057532207,test-aurora-master-instance,rdsadmin,localhost,3,3929,QUERY,mysql,'INSERT INTO mysql.rds_heartbeat2(id, value) values (1,1588934057440) ON DUPLICATE KEY UPDATE value = 1588934057440',0"},{"id":"35434413555180559722498853891203907271650676940633997315","timestamp":1588934057572,"message":"1588934057572920,test-aurora-master-instance,rdsadmin,localhost,3,3931,QUERY,mysql,'COMMIT',0"}]}{"messageType":"DATA_MESSAGE","owner":"XXXXXXXXXXXX","logGroup":"/aws/rds/cluster/test-aurora-cluster/audit","logStream":"test-aurora-master-instance.audit.log.1.2020-05-08-10-22.0.3","subscriptionFilters":["Destination"],"logEvents":[{"id":"35434413596124727907001077981448887272857346425521045504","timestamp":1588934059408,"message":"1588934059408077,test-aurora-master-instance,rdsadmin,localhost,3,3936,QUERY,mysql,'SET @@sql_log_bin=off',0"},{"id":"35434413598287900191258548426177851945304237491601145857","timestamp":1588934059505,"message":"1588934059505436,test-aurora-master-instance,rdsadmin,localhost,3,3938,QUERY,mysql,'INSERT INTO mysql.rds_heartbeat2(id, value) values (1,1588934059408) ON DUPLICATE KEY UPDATE value = 1588934059408',0"},{"id":"35434413598377103172052670918743994818394830937625067522","timestamp":1588934059509,"message":"1588934059509389,test-aurora-master-instance,rdsadmin,localhost,3,3939,QUERY,mysql,'select @@session.tx_read_only',0"},{"id":"35434413609728182478104758097785675419172846944169099267","timestamp":1588934060018,"message":"1588934060018177,test-aurora-master-instance,rdsadmin,localhost,2,3942,QUERY,mysql,'set local oscar_local_only_replica_host_status=1',0"}]}{"messageType":"DATA_MESSAGE","owner":"XXXXXXXXXXXX","logGroup":"/aws/rds/cluster/test-aurora-cluster/audit","logStream":"test-aurora-master-instance.audit.log.0.2020-05-08-10-22.0.1","subscriptionFilters":["Destination"],"logEvents":[{"id":"35434413609728182478104758097827802856977073558479437824","timestamp":1588934060018,"message":"1588934060018596,test-aurora-master-instance,rdsadmin,localhost,2,3943,QUERY,mysql,'SELECT durable_lsn, current_read_point, server_id, last_update_timestamp FROM information_schema.replica_host_status',0"},{"id":"35434413609750483223303288720969338575249721919985418241","timestamp":1588934060019,"message":"1588934060019019,test-aurora-master-instance,rdsadmin,localhost,2,3944,QUERY,mysql,'set local oscar_local_only_replica_host_status=0',0"}]}{"messageType":"DATA_MESSAGE","owner":"XXXXXXXXXXXX","logGroup":"/aws/rds/cluster/test-aurora-cluster/audit","logStream":"test-aurora-master-instance.audit.log.2.2020-05-08-10-22.0.1","subscriptionFilters":["Destination"],"logEvents":[{"id":"35434413596749148772559935429486627283808411669612593152","timestamp":1588934059436,"message":"1588934059436857,test-aurora-master-instance,rdsadmin,localhost,3,3937,QUERY,mysql,'select @@session.tx_read_only',0"},{"id":"35434413599023824782810058989923270547618544543222595585","timestamp":1588934059538,"message":"1588934059538397,test-aurora-master-instance,rdsadmin,localhost,3,3940,QUERY,mysql,'COMMIT',0"}]}{"messageType":"DATA_MESSAGE","owner":"XXXXXXXXXXXX","logGroup":"/aws/rds/cluster/test-aurora-cluster/audit","logStream":"test-aurora-master-instance.audit.log.3.2020-05-08-10-22.0.1","subscriptionFilters":["Destination"],"logEvents":[{"id":"35434413598131794974868834064295477414293791931759067136","timestamp":1588934059498,"message":"1588934059498579,test-aurora-master-instance,rdsadmin,localhost,3,3938,WRITE,mysql,rds_heartbeat2,"},{"id":"35434413600406470985118957624732120678103924805369069569","timestamp":1588934059600,"message":"1588934059600402,test-aurora-master-instance,rdsadmin,localhost,3,3941,QUERY,mysql,'SET @@sql_log_bin=on',0"}]}{"messageType":"DATA_MESSAGE","owner":"XXXXXXXXXXXX","logGroup":"/aws/rds/cluster/test-aurora-cluster/audit","logStream":"test-aurora-master-instance.audit.log.3.2020-05-08-10-22.0.2","subscriptionFilters":["Destination"],"logEvents":[{"id":"35434413644718051694599305809262979067530847728914268160","timestamp":1588934061587,"message":"1588934061587770,test-aurora-master-instance,rdsadmin,localhost,3,3950,QUERY,mysql,'SET @@sql_log_bin=on',0"}]}{"messageType":"DATA_MESSAGE","owner":"XXXXXXXXXXXX","logGroup":"/aws/rds/cluster/test-aurora-cluster/audit","logStream":"test-aurora-master-instance.audit.log.2.2020-05-08-10-22.0.3","subscriptionFilters":["Destination"],"logEvents":[{"id":"35434413640726218304062324266972944487195773308452536320","timestamp":1588934061408,"message":"1588934061408920,test-aurora-master-instance,rdsadmin,localhost,3,3945,QUERY,mysql,'SET @@sql_log_bin=off',0"},{"id":"35434413642086563761172692278606623301827323360317341697","timestamp":1588934061469,"message":"1588934061469113,test-aurora-master-instance,rdsadmin,localhost,3,3946,QUERY,mysql,'select @@session.tx_read_only',0"},{"id":"35434413642287270467959467886880444766281158613871165442","timestamp":1588934061478,"message":"1588934061478366,test-aurora-master-instance,rdsadmin,localhost,3,3947,WRITE,mysql,rds_heartbeat2,"},{"id":"35434413642398774193952121002588123357644400421401067523","timestamp":1588934061483,"message":"1588934061483169,test-aurora-master-instance,rdsadmin,localhost,3,3947,QUERY,mysql,'INSERT INTO mysql.rds_heartbeat2(id, value) values (1,1588934061409) ON DUPLICATE KEY UPDATE value = 1588934061409',0"},{"id":"35434413643000894314312447827409587751005906182062538756","timestamp":1588934061510,"message":"1588934061510046,test-aurora-master-instance,rdsadmin,localhost,3,3948,QUERY,mysql,'select @@session.tx_read_only',0"},{"id":"35434413644338939026224285215901730847364807872421363717","timestamp":1588934061570,"message":"1588934061570485,test-aurora-master-instance,rdsadmin,localhost,3,3949,QUERY,mysql,'COMMIT',0"},{"id":"35434413654485778091555718745300482661419812357642452998","timestamp":1588934062025,"message":"1588934062025241,test-aurora-master-instance,rdsadmin,localhost,2,3951,QUERY,mysql,'set local oscar_local_only_replica_host_status=1',0"},{"id":"35434413654485778091555718745300482661419812357642452999","timestamp":1588934062025,"message":"1588934062025653,test-aurora-master-instance,rdsadmin,localhost,2,3952,QUERY,mysql,'SELECT durable_lsn, current_read_point, server_id, last_update_timestamp FROM information_schema.replica_host_status',0"},{"id":"35434413654508078836754249368442018379692460719148433416","timestamp":1588934062026,"message":"1588934062026523,test-aurora-master-instance,rdsadmin,localhost,2,3953,QUERY,mysql,'set local oscar_local_only_replica_host_status=0',0"}]}{"messageType":"DATA_MESSAGE","owner":"XXXXXXXXXXXX","logGroup":"/aws/rds/cluster/test-aurora-cluster/audit","logStream":"test-aurora-master-instance.audit.log.3.2020-05-08-10-22.0.1","subscriptionFilters":["Destination"],"logEvents":[{"id":"35434413687357076514189857258419340990260553885336535040","timestamp":1588934063499,"message":"1588934063499128,test-aurora-master-instance,rdsadmin,localhost,3,3956,QUERY,mysql,'INSERT INTO mysql.rds_heartbeat2(id, value) values (1,1588934063415) ON DUPLICATE KEY UPDATE value = 1588934063415',0"},{"id":"35434413687557783220976632866693162454714389138890358785","timestamp":1588934063508,"message":"1588934063508476,test-aurora-master-instance,rdsadmin,localhost,3,3957,QUERY,mysql,'select @@session.tx_read_only',0"},{"id":"35434413689118835384873776486600662733799774444308987906","timestamp":1588934063578,"message":"1588934063578280,test-aurora-master-instance,rdsadmin,localhost,3,3959,QUERY,mysql,'SET @@sql_log_bin=on',0"}]}{"messageType":"DATA_MESSAGE","owner":"XXXXXXXXXXXX","logGroup":"/aws/rds/cluster/test-aurora-cluster/audit","logStream":"test-aurora-master-instance.audit.log.1.2020-05-08-10-22.0.2","subscriptionFilters":["Destination"],"logEvents":[{"id":"35434413687200971297800142896316481564120061462605987840","timestamp":1588934063492,"message":"1588934063492923,test-aurora-master-instance,rdsadmin,localhost,3,3956,WRITE,mysql,rds_heartbeat2,"},{"id":"35434413688070700360542837198836374576753347561339224065","timestamp":1588934063531,"message":"1588934063531865,test-aurora-master-instance,rdsadmin,localhost,3,3958,QUERY,mysql,'COMMIT',0"},{"id":"35434413698953464017425781291905805093805747976257667074","timestamp":1588934064019,"message":"1588934064019907,test-aurora-master-instance,rdsadmin,localhost,2,3960,QUERY,mysql,'set local oscar_local_only_replica_host_status=1',0"},{"id":"35434413699109569233815495653896555121714286506799529987","timestamp":1588934064026,"message":"1588934064026525,test-aurora-master-instance,rdsadmin,localhost,2,3962,QUERY,mysql,'set local oscar_local_only_replica_host_status=0',0"}]}{"messageType":"DATA_MESSAGE","owner":"XXXXXXXXXXXX","logGroup":"/aws/rds/cluster/test-aurora-cluster/audit","logStream":"test-aurora-master-instance.audit.log.0.2020-05-08-10-22.0.0","subscriptionFilters":["Destination"],"logEvents":[{"id":"35434413685461513172314754291349038010220939088768991232","timestamp":1588934063414,"message":"1588934063414999,test-aurora-master-instance,rdsadmin,localhost,3,3954,QUERY,mysql,'SET @@sql_log_bin=off',0"}]}{"messageType":"DATA_MESSAGE","owner":"XXXXXXXXXXXX","logGroup":"/aws/rds/cluster/test-aurora-cluster/audit","logStream":"test-aurora-master-instance.audit.log.2.2020-05-08-10-22.0.2","subscriptionFilters":["Destination"],"logEvents":[{"id":"35434413686866460119822183549234056318438162316015042560","timestamp":1588934063477,"message":"1588934063477661,test-aurora-master-instance,rdsadmin,localhost,3,3955,QUERY,mysql,'select @@session.tx_read_only',0"},{"id":"35434413699087268488616965030795629931849464421292310529","timestamp":1588934064025,"message":"1588934064025432,test-aurora-master-instance,rdsadmin,localhost,2,3961,QUERY,mysql,'SELECT durable_lsn, current_read_point, server_id, last_update_timestamp FROM information_schema.replica_host_status',0"}]}{"messageType":"DATA_MESSAGE","owner":"XXXXXXXXXXXX","logGroup":"/aws/rds/cluster/test-aurora-cluster/audit","logStream":"test-aurora-master-instance.audit.log.1.2020-05-08-10-22.0.2","subscriptionFilters":["Destination"],"logEvents":[{"id":"35434413730665123689736327401583556463662760122664681472","timestamp":1588934065441,"message":"1588934065441345,test-aurora-master-instance,rdsadmin,localhost,3,3963,QUERY,mysql,'SET @@sql_log_bin=off',0"},{"id":"35434413731401048281287837965254235166660156052362035201","timestamp":1588934065474,"message":"1588934065474085,test-aurora-master-instance,rdsadmin,localhost,3,3964,QUERY,mysql,'select @@session.tx_read_only',0"},{"id":"35434413731646356478471674819811128067659288028927819778","timestamp":1588934065485,"message":"1588934065485970,test-aurora-master-instance,rdsadmin,localhost,3,3965,WRITE,mysql,rds_heartbeat2,"},{"id":"35434413731780160949662858558660342377295178197963702275","timestamp":1588934065491,"message":"1588934065491177,test-aurora-master-instance,rdsadmin,localhost,3,3965,QUERY,mysql,'INSERT INTO mysql.rds_heartbeat2(id, value) values (1,1588934065441) ON DUPLICATE KEY UPDATE value = 1588934065441',0"},{"id":"35434413733296611623162940932284771219835266780370370564","timestamp":1588934065559,"message":"1588934065559229,test-aurora-master-instance,rdsadmin,localhost,3,3967,QUERY,mysql,'COMMIT',0"},{"id":"35434413744736893910009150603892594693703876232938323973","timestamp":1588934066072,"message":"1588934066072701,test-aurora-master-instance,rdsadmin,localhost,2,3969,QUERY,mysql,'set local oscar_local_only_replica_host_status=1',0"},{"id":"35434413744759194655207681227034130411976524594444304390","timestamp":1588934066073,"message":"1588934066073148,test-aurora-master-instance,rdsadmin,localhost,2,3970,QUERY,mysql,'SELECT durable_lsn, current_read_point, server_id, last_update_timestamp FROM information_schema.replica_host_status',0"},{"id":"35434413744781495400406211850175666130249172955950284807","timestamp":1588934066074,"message":"1588934066074026,test-aurora-master-instance,rdsadmin,localhost,2,3971,QUERY,mysql,'set local oscar_local_only_replica_host_status=0',0"}]}{"messageType":"DATA_MESSAGE","owner":"XXXXXXXXXXXX","logGroup":"/aws/rds/cluster/test-aurora-cluster/audit","logStream":"test-aurora-master-instance.audit.log.3.2020-05-08-10-22.0.2","subscriptionFilters":["Destination"],"logEvents":[{"id":"35434413731936266166052572920690748225219117265405542400","timestamp":1588934065498,"message":"1588934065498168,test-aurora-master-instance,rdsadmin,localhost,3,3966,QUERY,mysql,'select @@session.tx_read_only',0"},{"id":"35434413733631122801140900279447462813940392739860054017","timestamp":1588934065574,"message":"1588934065574585,test-aurora-master-instance,rdsadmin,localhost,3,3968,QUERY,mysql,'SET @@sql_log_bin=on',0"}]}{"messageType":"DATA_MESSAGE","owner":"XXXXXXXXXXXX","logGroup":"/aws/rds/cluster/test-aurora-cluster/audit","logStream":"test-aurora-master-instance.audit.log.1.2020-05-08-10-22.0.1","subscriptionFilters":["Destination"],"logEvents":[{"id":"35434413769245412883194305437654139856525769608358395904","timestamp":1588934067171,"message":"1588934067171724,test-aurora-master-instance,rdsadmin,localhost,39,3972,QUERY,mysql,'SELECT VARIABLE_NAME, VARIABLE_VALUE FROM INFORMATION_SCHEMA.GLOBAL_STATUS WHERE VARIABLE_NAME IN (\\'QCACHE_HITS\\', \\'COM_SELECT\\', \\'INNODB_BUFFER_POOL_READS\\', \\'INNODB_BUFFER_POOL_READ_REQUESTS\\')',0"},{"id":"35434413769936735984348754755041747122977868815043788801","timestamp":1588934067202,"message":"1588934067202374,test-aurora-master-instance,rdsadmin,localhost,39,3973,QUERY,mysql,'SELECT VARIABLE_NAME, VARIABLE_VALUE FROM INFORMATION_SCHEMA.GLOBAL_STATUS WHERE VARIABLE_NAME IN ( \\'COM_DROP_TRIGGER\\', \\'COM_DROP_TABLE\\', \\'AURORADB_SELECT_STMT_DURATION\\', \\'COM_SELECT\\', \\'AURORA_LOCKMGR_MEMORY_USED\\', \\'COM_CREATE_UDF\\', \\'COM_DROP_FUNCTION\\', \\'COM_ALTER_EVENT\\', \\'COM_CREATE_INDEX\\', \\'COM_CREATE_PROCEDURE\\', \\'HANDLER_UPDATE\\', \\'COM_ALTER_TABLE\\', \\'HANDLER_WRITE\\', \\'COM_DELETE_MULTI\\', \\'AURORADB_DELETE_STMT_DURATION\\', \\'COM_TRUNCATE\\', \\'COM_ALTER_SERVER\\', \\'COM_DROP_PROCEDURE\\', \\'COM_CREATE_TABLE\\', \\'COM_CREATE_DB\\', \\'COUNT|COM_INSERT|COM_INSERT_SELECT\\', \\'HANDLER_DELETE\\', \\'QCACHE_HITS\\', \\'COM_CREATE_EVENT\\', \\'COM_DELETE\\', \\'COM_CREATE_FUNCTION\\', \\'COM_ALTER_SYSTEM\\', \\'AURORADB_COMMIT_LATENCY\\', \\'COM_UPDATE_MULTI\\', \\'COUNT|COM_UPDATE|COM_UPDATE_MULTI\\', \\'AURORADB_INSERT_STMT_DURATION\\', \\'COM_ALTER_TABLESPACE\\', \\'COM_DROP_DB\\', \\'COM_DROP_SERVER\\', \\'COM_INSERT\\', \\'AURORADB_DDL_STMT_DURATION\\', \\'AURORA_ROLLBACK_SEGMENT_HISTORY_LENGTH\\', \\'AURORADB_UPDATE_STMT_DURATION\\', \\'COM_ALTER_DB\\', \\'COM_ALTER_PROCEDURE\\', \\'COM_ALTER_DB_UPGRADE\\', \\'COM_INSERT_SELECT\\', \\'COM_CREATE_SERVER\\', \\'COM_CREATE_TRIGGER\\', \\'COUNT|COM_DELETE|COM_DELETE_MULTI|COM_TRUNCATE\\', \\'AURORADB_COMMITS\\', \\'COM_DROP_INDEX\\', \\'COM_UPDATE\\', \\'COM_ALTER_FUNCTION\\', \\'COM_CREATE_VIEW\\', \\'COM_DROP_EVENT\\', \\'COM_DROP_VIEW\\')',0"},{"id":"35434413771809998581025327098930747457880331181546143746","timestamp":1588934067286,"message":"1588934067286916,test-aurora-master-instance,rdsadmin,localhost,39,3974,QUERY,mysql,'SELECT VARIABLE_NAME, VARIABLE_VALUE FROM INFORMATION_SCHEMA.GLOBAL_STATUS WHERE VARIABLE_NAME IN ( \\'ABORTED_CONNECTS\\', \\'QUERIES\\', \\'THREADS_CONNECTED\\')',0"},{"id":"35434413771899201561819449591496890330970924627570065411","timestamp":1588934067290,"message":"1588934067290934,test-aurora-master-instance,rdsadmin,localhost,39,3975,QUERY,mysql,'SELECT count(*) from information_schema.TABLES WHERE TABLE_SCHEMA = \\'information_schema\\' AND TABLE_NAME = \\'RDS_METRICS_META\\'',0"},{"id":"35434413772545923172576837662601426160877727111243497476","timestamp":1588934067319,"message":"1588934067319098,test-aurora-master-instance,rdsadmin,localhost,39,3976,QUERY,mysql,'SELECT NAME, COUNT FROM INFORMATION_SCHEMA.INNODB_METRICS WHERE NAME IN (\\'LOCK_DEADLOCKS\\', \\'LOCK_ROW_LOCK_CURRENT_WAITS\\', \\'TRX_ACTIVE_TRANSACTIONS\\')',0"}]}{"messageType":"DATA_MESSAGE","owner":"XXXXXXXXXXXX","logGroup":"/aws/rds/cluster/test-aurora-cluster/audit","logStream":"test-aurora-master-instance.audit.log.1.2020-05-08-10-22.0.0","subscriptionFilters":["Destination"],"logEvents":[{"id":"35434413773594058196907776951453984271822081557466382336","timestamp":1588934067366,"message":"1588934067366092,test-aurora-master-instance,rdsadmin,localhost,39,3977,QUERY,mysql,'SELECT @@GLOBAL.aurora_backtrack_enabled',1193"},{"id":"35434413774106975336473981283709305792092993872103931905","timestamp":1588934067389,"message":"1588934067389761,test-aurora-master-instance,rdsadmin,localhost,39,0,DISCONNECT,,,0"},{"id":"35434413774151576826871042529992377228638290595115892738","timestamp":1588934067391,"message":"1588934067391495,test-aurora-master-instance,rdsadmin,localhost,41,0,CONNECT,,,0"},{"id":"35434413774173877572069573153133912946910938956621873155","timestamp":1588934067392,"message":"1588934067392207,test-aurora-master-instance,rdsadmin,localhost,41,3978,QUERY,,'SET SESSION wait_timeout=28800',0"},{"id":"35434413774865200673224022470521520213363038163307266052","timestamp":1588934067423,"message":"1588934067423081,test-aurora-master-instance,rdsadmin,localhost,41,3979,QUERY,,'SET SESSION time_zone=UTC',0"},{"id":"35434413774865200673224022470521520213363038163307266053","timestamp":1588934067423,"message":"1588934067423111,test-aurora-master-instance,rdsadmin,localhost,3,3980,QUERY,mysql,'SET @@sql_log_bin=off',0"},{"id":"35434413776403952091922635467287484774175775107219914758","timestamp":1588934067492,"message":"1588934067492948,test-aurora-master-instance,rdsadmin,localhost,3,3982,QUERY,mysql,'select @@session.tx_read_only',0"},{"id":"35434413776403952091922635467287484774175775107219914759","timestamp":1588934067492,"message":"1588934067492966,test-aurora-master-instance,rdsadmin,localhost,41,3981,QUERY,,'SET SESSION sql_mode=0',0"},{"id":"35434413776649260289106472321844377675174907083785699336","timestamp":1588934067503,"message":"1588934067503826,test-aurora-master-instance,rdsadmin,localhost,3,3984,WRITE,mysql,rds_heartbeat2,"},{"id":"35434413776649260289106472321844377675174907083785699337","timestamp":1588934067503,"message":"1588934067503966,test-aurora-master-instance,rdsadmin,localhost,41,3983,QUERY,,'SET SESSION autocommit=1',0"},{"id":"35434413776983771467084431668967413449264632506375405578","timestamp":1588934067518,"message":"1588934067518936,test-aurora-master-instance,rdsadmin,localhost,3,3984,QUERY,mysql,'INSERT INTO mysql.rds_heartbeat2(id, value) values (1,1588934067423) ON DUPLICATE KEY UPDATE value = 1588934067423',0"},{"id":"35434413777162177428672676654099699195445819398423248907","timestamp":1588934067526,"message":"1588934067526579,test-aurora-master-instance,rdsadmin,localhost,41,3985,QUERY,,'/* mysql-connector-java-5.1.33 ( Revision: [email protected] ) */SHOW VARIABLES WHERE Variable_name =\\'language\\' OR Variable_name = \\'net_write_timeout\\' OR Variable_name = \\'interactive_timeout\\' OR Variable_name = \\'wait_timeout\\' OR Variable_name = \\'character_set_client\\' OR Variable_name = \\'character_set_connection\\' OR Variable_name = \\'character_set\\' OR Variable_name = \\'character_set_server\\' OR Variable_name = \\'tx_isolation\\' OR Variable_name = \\'transaction_isolation\\' OR Variable_name = \\'character_set_results\\' OR Variable_name = \\'timezone\\' OR Variable_name = \\'time_zone\\' OR Variable_name = \\'system_time_zone\\' OR Variable_name = \\'lower_case_table_names\\' OR Variable_name = \\'max_allowed_packet\\' OR Variable_name = \\'net_buffer_length\\' OR Variable_name = \\'sql_mode\\' OR Variable_name = \\'query_cache_type\\' OR Variable_name = \\'query_cache_size\\' OR Variable_name = \\'license\\' OR Variable_name = \\'init_connect\\'',0"},{"id":"35434413777184478173871207277241234913718467759929229324","timestamp":1588934067527,"message":"1588934067527499,test-aurora-master-instance,rdsadmin,localhost,3,3986,QUERY,mysql,'select @@session.tx_read_only',0"},{"id":"35434413778656327356974228404582592319713259619323936781","timestamp":1588934067593,"message":"1588934067593520,test-aurora-master-instance,rdsadmin,localhost,41,3987,QUERY,,'/* mysql-connector-java-5.1.33 ( Revision: [email protected] ) */SELECT @@session.auto_increment_increment',0"},{"id":"35434413778656327356974228404582592319713259619323936782","timestamp":1588934067593,"message":"1588934067593641,test-aurora-master-instance,rdsadmin,localhost,3,3988,QUERY,mysql,'COMMIT',0"},{"id":"35434413779481454929319861460819413895801248995045212175","timestamp":1588934067630,"message":"1588934067630609,test-aurora-master-instance,rdsadmin,localhost,41,3991,QUERY,,'SET character_set_results = NULL',0"}]}{"messageType":"DATA_MESSAGE","owner":"XXXXXXXXXXXX","logGroup":"/aws/rds/cluster/test-aurora-cluster/audit","logStream":"test-aurora-master-instance.audit.log.2.2020-05-08-10-22.0.0","subscriptionFilters":["Destination"],"logEvents":[{"id":"35434413778968537789753657128605118322998689825194901504","timestamp":1588934067607,"message":"1588934067607235,test-aurora-master-instance,rdsadmin,localhost,41,3989,QUERY,,'SET NAMES utf8',0"},{"id":"35434413778968537789753657128605118322998689825194901505","timestamp":1588934067607,"message":"1588934067607252,test-aurora-master-instance,rdsadmin,localhost,3,3990,QUERY,mysql,'SET @@sql_log_bin=on',0"},{"id":"35434413780908702622025821341918725812719097276215197698","timestamp":1588934067694,"message":"1588934067694384,test-aurora-master-instance,rdsadmin,localhost,41,3992,QUERY,,'SET autocommit=1',0"},{"id":"35434413780931003367224351965060261530991745637721178115","timestamp":1588934067695,"message":"1588934067695326,test-aurora-master-instance,rdsadmin,localhost,41,3993,QUERY,,'SET sql_mode=\\'STRICT_TRANS_TABLES\\'',0"},{"id":"35434413781198612309606719442758690150263525975792943108","timestamp":1588934067707,"message":"1588934067707668,test-aurora-master-instance,rdsadmin,localhost,41,3994,QUERY,,'SHOW WARNINGS',0"},{"id":"35434413781733830194371454398155547388807086651936473093","timestamp":1588934067731,"message":"1588934067731171,test-aurora-master-instance,rdsadmin,localhost,41,3995,QUERY,,'set local oscar_local_only_replica_host_status=1',0"},{"id":"35434413781733830194371454398155547388807086651936473094","timestamp":1588934067731,"message":"1588934067731705,test-aurora-master-instance,rdsadmin,localhost,41,3996,QUERY,,'SELECT MIN(REPLICA_LAG_IN_MILLISECONDS), MAX(REPLICA_LAG_IN_MILLISECONDS) FROM INFORMATION_SCHEMA.REPLICA_HOST_STATUS WHERE SERVER_ID != \\'test-aurora-master-instance\\'',0"},{"id":"35434413781756130939569985021297083107079735013442453511","timestamp":1588934067732,"message":"1588934067732690,test-aurora-master-instance,rdsadmin,localhost,41,3997,QUERY,,'set local oscar_local_only_replica_host_status=0',0"},{"id":"35434413788580158970320355702607012898510133634272460808","timestamp":1588934068038,"message":"1588934068038624,test-aurora-master-instance,rdsadmin,localhost,2,3998,QUERY,mysql,'set local oscar_local_only_replica_host_status=1',0"},{"id":"35434413788602459715518886325748548616782781995778441225","timestamp":1588934068039,"message":"1588934068039091,test-aurora-master-instance,rdsadmin,localhost,2,3999,QUERY,mysql,'SELECT durable_lsn, current_read_point, server_id, last_update_timestamp FROM information_schema.replica_host_status',0"},{"id":"35434413788602459715518886325748548616782781995778441226","timestamp":1588934068039,"message":"1588934068039901,test-aurora-master-instance,rdsadmin,localhost,2,4000,QUERY,mysql,'set local oscar_local_only_replica_host_status=0',0"}]}{"messageType":"DATA_MESSAGE","owner":"XXXXXXXXXXXX","logGroup":"/aws/rds/cluster/test-aurora-cluster/audit","logStream":"test-aurora-master-instance.audit.log.1.2020-05-08-10-22.0.1","subscriptionFilters":["Destination"],"logEvents":[{"id":"35434413820001908955050003711402772527407830613708701696","timestamp":1588934069447,"message":"1588934069447357,test-aurora-master-instance,rdsadmin,localhost,3,4002,QUERY,mysql,'select @@session.tx_read_only',0"},{"id":"35434413820737833546601514275073451230405226543406055425","timestamp":1588934069480,"message":"1588934069480313,test-aurora-master-instance,rdsadmin,localhost,3,4003,QUERY,mysql,'INSERT INTO mysql.rds_heartbeat2(id, value) values (1,1588934069428) ON DUPLICATE KEY UPDATE value = 1588934069428',0"},{"id":"35434413835701633574815562403043918191352277113918914562","timestamp":1588934070151,"message":"1588934070151579,test-aurora-master-instance,rdsadmin,localhost,2,4008,QUERY,mysql,'set local oscar_local_only_replica_host_status=1',0"},{"id":"35434413835723934320014093026185453909624925475424894979","timestamp":1588934070152,"message":"1588934070152041,test-aurora-master-instance,rdsadmin,localhost,2,4009,QUERY,mysql,'SELECT durable_lsn, current_read_point, server_id, last_update_timestamp FROM information_schema.replica_host_status',0"},{"id":"35434413835723934320014093026185453909624925475424894980","timestamp":1588934070152,"message":"1588934070152261,test-aurora-master-instance,rdsadmin,localhost,2,4010,QUERY,mysql,'set local oscar_local_only_replica_host_status=0',0"},{"id":"35434413835924641026800868634459275374078760728978718725","timestamp":1588934070161,"message":"1588934070161738,test-aurora-master-instance,rdsadmin,localhost,2,4011,QUERY,mysql,'set local oscar_local_only_replica_host_status=1',0"},{"id":"35434413836013844007594991127025418247169354175002640390","timestamp":1588934070165,"message":"1588934070165631,test-aurora-master-instance,rdsadmin,localhost,2,4012,QUERY,mysql,'SELECT durable_lsn, current_read_point, server_id, last_update_timestamp FROM information_schema.replica_host_status',0"},{"id":"35434413836036144752793521750166953965442002536508620807","timestamp":1588934070166,"message":"1588934070166059,test-aurora-master-instance,rdsadmin,localhost,2,4013,QUERY,mysql,'set local oscar_local_only_replica_host_status=0',0"}]}{"messageType":"DATA_MESSAGE","owner":"XXXXXXXXXXXX","logGroup":"/aws/rds/cluster/test-aurora-cluster/audit","logStream":"test-aurora-master-instance.audit.log.2.2020-05-08-10-22.0.3","subscriptionFilters":["Destination"],"logEvents":[{"id":"35434413819578194796277921871760822489951285639401111552","timestamp":1588934069428,"message":"1588934069428514,test-aurora-master-instance,rdsadmin,localhost,3,4001,QUERY,mysql,'SET @@sql_log_bin=off',0"},{"id":"35434413821830570061329514809055930035488770151505133569","timestamp":1588934069529,"message":"1588934069529719,test-aurora-master-instance,rdsadmin,localhost,3,4004,QUERY,mysql,'select @@session.tx_read_only',0"},{"id":"35434413822165081239307474156178965809578495574094839810","timestamp":1588934069544,"message":"1588934069544950,test-aurora-master-instance,rdsadmin,localhost,3,4005,QUERY,mysql,'COMMIT',0"},{"id":"35434413822254284220101596648745108682669089020118761475","timestamp":1588934069548,"message":"1588934069548198,test-aurora-master-instance,rdsadmin,localhost,3,4006,QUERY,mysql,'SET @@sql_log_bin=on',0"},{"id":"35434413833806070232940459436060610747900940280216616964","timestamp":1588934070066,"message":"1588934070066556,test-aurora-master-instance,rdsadmin,localhost,41,4007,QUERY,,'SELECT @@aurora_version',0"}]}{"messageType":"DATA_MESSAGE","owner":"XXXXXXXXXXXX","logGroup":"/aws/rds/cluster/test-aurora-cluster/audit","logStream":"test-aurora-master-instance.audit.log.3.2020-05-08-10-22.0.3","subscriptionFilters":["Destination"],"logEvents":[{"id":"35434413820537126839814738666890382724404641301009661952","timestamp":1588934069471,"message":"1588934069471879,test-aurora-master-instance,rdsadmin,localhost,3,4003,WRITE,mysql,rds_heartbeat2,"}]}{"messageType":"DATA_MESSAGE","owner":"XXXXXXXXXXXX","logGroup":"/aws/rds/cluster/test-aurora-cluster/audit","logStream":"test-aurora-master-instance.audit.log.2.2020-05-08-10-22.0.0","subscriptionFilters":["Destination"],"logEvents":[{"id":"35434413863755971034567086317579035689746175668805894144","timestamp":1588934071409,"message":"1588934071409565,test-aurora-master-instance,rdsadmin,localhost,3,4014,QUERY,mysql,'SET @@sql_log_bin=off',0"},{"id":"35434413865740737357236311777175714616011879842838151169","timestamp":1588934071498,"message":"1588934071498336,test-aurora-master-instance,rdsadmin,localhost,3,4017,QUERY,mysql,'select @@session.tx_read_only',0"},{"id":"35434413866989579088354026673101714839280188087173054466","timestamp":1588934071554,"message":"1588934071554903,test-aurora-master-instance,rdsadmin,localhost,3,4019,QUERY,mysql,'SET @@sql_log_bin=on',0"},{"id":"35434413878519064355994358837275681186239390985764929539","timestamp":1588934072071,"message":"1588934072071488,test-aurora-master-instance,rdsadmin,localhost,2,4021,QUERY,mysql,'SELECT durable_lsn, current_read_point, server_id, last_update_timestamp FROM information_schema.replica_host_status',0"},{"id":"35434413878541365101192889460417216904512039347270909956","timestamp":1588934072072,"message":"1588934072072051,test-aurora-master-instance,rdsadmin,localhost,2,4022,QUERY,mysql,'set local oscar_local_only_replica_host_status=0',0"}]}{"messageType":"DATA_MESSAGE","owner":"XXXXXXXXXXXX","logGroup":"/aws/rds/cluster/test-aurora-cluster/audit","logStream":"test-aurora-master-instance.audit.log.1.2020-05-08-10-22.0.0","subscriptionFilters":["Destination"],"logEvents":[{"id":"35434413864737203823302433735758223400004703927838179328","timestamp":1588934071453,"message":"1588934071453937,test-aurora-master-instance,rdsadmin,localhost,3,4015,QUERY,mysql,'select @@session.tx_read_only',0"},{"id":"35434413865183218727273046198588937765457671157957787649","timestamp":1588934071473,"message":"1588934071473986,test-aurora-master-instance,rdsadmin,localhost,3,4016,WRITE,mysql,rds_heartbeat2,"},{"id":"35434413866030647044817209877967295059818308895185043458","timestamp":1588934071511,"message":"1588934071511476,test-aurora-master-instance,rdsadmin,localhost,3,4018,QUERY,mysql,'COMMIT',0"},{"id":"35434413878496763610795828214085761574228742977028096003","timestamp":1588934072070,"message":"1588934072070774,test-aurora-master-instance,rdsadmin,localhost,2,4020,QUERY,mysql,'set local oscar_local_only_replica_host_status=1',0"}]}{"messageType":"DATA_MESSAGE","owner":"XXXXXXXXXXXX","logGroup":"/aws/rds/cluster/test-aurora-cluster/audit","logStream":"test-aurora-master-instance.audit.log.3.2020-05-08-10-22.0.2","subscriptionFilters":["Destination"],"logEvents":[{"id":"35434413865361624688861291183838412292959795057159503872","timestamp":1588934071481,"message":"1588934071481147,test-aurora-master-instance,rdsadmin,localhost,3,4016,QUERY,mysql,'INSERT INTO mysql.rds_heartbeat2(id, value) values (1,1588934071409) ON DUPLICATE KEY UPDATE value = 1588934071409',0"}]}{"messageType":"DATA_MESSAGE","owner":"XXXXXXXXXXXX","logGroup":"/aws/rds/cluster/test-aurora-cluster/audit","logStream":"test-aurora-master-instance.audit.log.0.2020-05-08-10-22.0.1","subscriptionFilters":["Destination"],"logEvents":[{"id":"35434413886480430391869791298839337726215486229462712320","timestamp":1588934072428,"message":"1588934072428328,test-aurora-master-instance,rdsadmin,localhost,41,4026,QUERY,,'SELECT @@GLOBAL.aurora_backtrack_enabled',1193"}]}{"messageType":"DATA_MESSAGE","owner":"XXXXXXXXXXXX","logGroup":"/aws/rds/cluster/test-aurora-cluster/audit","logStream":"test-aurora-master-instance.audit.log.2.2020-05-08-10-22.0.0","subscriptionFilters":["Destination"],"logEvents":[{"id":"35434413885811408035913872605723514152766791601521819648","timestamp":1588934072398,"message":"1588934072398508,test-aurora-master-instance,rdsadmin,localhost,2,4023,QUERY,mysql,'set local oscar_local_only_replica_host_status=1',0"},{"id":"35434413886859543060244811893375692911581264592302899201","timestamp":1588934072445,"message":"1588934072445508,test-aurora-master-instance,rdsadmin,localhost,41,0,DISCONNECT,,,0"},{"id":"35434413888152986281759588035584764571394869559649763330","timestamp":1588934072503,"message":"1588934072503526,test-aurora-master-instance,rdsadmin,localhost,42,4028,QUERY,,'SET SESSION wait_timeout=28800',0"},{"id":"35434413889312625032083180438944621921572584357960744963","timestamp":1588934072555,"message":"1588934072555010,test-aurora-master-instance,rdsadmin,localhost,42,4030,QUERY,,'SET SESSION sql_mode=0',0"},{"id":"35434413889669436955259670409209193413934958142056431620","timestamp":1588934072571,"message":"1588934072571886,test-aurora-master-instance,rdsadmin,localhost,42,4031,QUERY,,'SET SESSION autocommit=1',0"},{"id":"35434413891765707003921548984513550931563904123618590725","timestamp":1588934072665,"message":"1588934072665489,test-aurora-master-instance,rdsadmin,localhost,42,4034,QUERY,,'SET NAMES utf8',0"},{"id":"35434413891944112965509793969645836677745091015666434054","timestamp":1588934072673,"message":"1588934072673610,test-aurora-master-instance,rdsadmin,localhost,42,4035,QUERY,,'SET character_set_results = NULL',0"},{"id":"35434413892702338302259835156458051099015135306869768199","timestamp":1588934072707,"message":"1588934072707141,test-aurora-master-instance,rdsadmin,localhost,42,4036,QUERY,,'SET autocommit=1',0"}]}{"messageType":"DATA_MESSAGE","owner":"XXXXXXXXXXXX","logGroup":"/aws/rds/cluster/test-aurora-cluster/audit","logStream":"test-aurora-master-instance.audit.log.1.2020-05-08-10-22.0.2","subscriptionFilters":["Destination"],"logEvents":[{"id":"35434413885833708781112403228833494720369653805265387520","timestamp":1588934072399,"message":"1588934072399026,test-aurora-master-instance,rdsadmin,localhost,2,4024,QUERY,mysql,'SELECT durable_lsn, current_read_point, server_id, last_update_timestamp FROM information_schema.replica_host_status',0"},{"id":"35434413885856009526310933851975030438642302166771367937","timestamp":1588934072400,"message":"1588934072400828,test-aurora-master-instance,rdsadmin,localhost,2,4025,QUERY,mysql,'set local oscar_local_only_replica_host_status=0',0"},{"id":"35434413887194054238222771240467173535001203857130192898","timestamp":1588934072460,"message":"1588934072460895,test-aurora-master-instance,rdsadmin,localhost,42,0,CONNECT,,,0"},{"id":"35434413890427662292009711595989852684535216275497353219","timestamp":1588934072605,"message":"1588934072605236,test-aurora-master-instance,rdsadmin,localhost,42,4032,QUERY,,'/* mysql-connector-java-5.1.33 ( Revision: [email protected] ) */SHOW VARIABLES WHERE Variable_name =\\'language\\' OR Variable_name = \\'net_write_timeout\\' OR Variable_name = \\'interactive_timeout\\' OR Variable_name = \\'wait_timeout\\' OR Variable_name = \\'character_set_client\\' OR Variable_name = \\'character_set_connection\\' OR Variable_name = \\'character_set\\' OR Variable_name = \\'character_set_server\\' OR Variable_name = \\'tx_isolation\\' OR Variable_name = \\'transaction_isolation\\' OR Variable_name = \\'character_set_results\\' OR Variable_name = \\'timezone\\' OR Variable_name = \\'time_zone\\' OR Variable_name = \\'system_time_zone\\' OR Variable_name = \\'lower_case_table_names\\' OR Variable_name = \\'max_allowed_packet\\' OR Variable_name = \\'net_buffer_length\\' OR Variable_name = \\'sql_mode\\' OR Variable_name = \\'query_cache_type\\' OR Variable_name = \\'query_cache_size\\' OR Variable_name = \\'license\\' OR Variable_name = \\'init_connect\\'',0"},{"id":"35434413894174187485362856283767853354340141008502063108","timestamp":1588934072773,"message":"1588934072773843,test-aurora-master-instance,rdsadmin,localhost,42,4038,QUERY,,'SHOW WARNINGS',0"},{"id":"35434413894241089720958448153192460509158086093020004357","timestamp":1588934072776,"message":"1588934072776495,test-aurora-master-instance,rdsadmin,localhost,42,4039,QUERY,,'SELECT @@GLOBAL.aurora_backtrack_enabled',1193"},{"id":"35434413894932412822112897470580067775610185299705397254","timestamp":1588934072807,"message":"1588934072807418,test-aurora-master-instance,rdsadmin,localhost,42,0,DISCONNECT,,,0"}]}{"messageType":"DATA_MESSAGE","owner":"XXXXXXXXXXXX","logGroup":"/aws/rds/cluster/test-aurora-cluster/audit","logStream":"test-aurora-master-instance.audit.log.0.2020-05-08-10-22.0.1","subscriptionFilters":["Destination"],"logEvents":[{"id":"35434413888732805656921384237298160903868078186608459776","timestamp":1588934072529,"message":"1588934072529465,test-aurora-master-instance,rdsadmin,localhost,42,4029,QUERY,,'SET SESSION time_zone=UTC',0"},{"id":"35434413890985180921972977174593268449405562698712481793","timestamp":1588934072630,"message":"1588934072630297,test-aurora-master-instance,rdsadmin,localhost,42,4033,QUERY,,'/* mysql-connector-java-5.1.33 ( Revision: [email protected] ) */SELECT @@session.auto_increment_increment',0"},{"id":"35434413893237556187024570111888375994943047210816503810","timestamp":1588934072731,"message":"1588934072731964,test-aurora-master-instance,rdsadmin,localhost,42,4037,QUERY,,'SET sql_mode=\\'STRICT_TRANS_TABLES\\'',0"}]}{"messageType":"DATA_MESSAGE","owner":"XXXXXXXXXXXX","logGroup":"/aws/rds/cluster/test-aurora-cluster/audit","logStream":"test-aurora-master-instance.audit.log.0.2020-05-08-10-22.0.1","subscriptionFilters":["Destination"],"logEvents":[{"id":"35434413908981882297187190051044750998031466019456221184","timestamp":1588934073437,"message":"1588934073437200,test-aurora-master-instance,rdsadmin,localhost,3,4041,QUERY,mysql,'SET @@sql_log_bin=off',0"}]}{"messageType":"DATA_MESSAGE","owner":"XXXXXXXXXXXX","logGroup":"/aws/rds/cluster/test-aurora-cluster/audit","logStream":"test-aurora-master-instance.audit.log.2.2020-05-08-10-22.0.0","subscriptionFilters":["Destination"],"logEvents":[{"id":"35434413909450197946356333136975865446866871589978505216","timestamp":1588934073458,"message":"1588934073458219,test-aurora-master-instance,rdsadmin,localhost,3,4042,QUERY,mysql,'select @@session.tx_read_only',0"},{"id":"35434413910342227754297558062637294177772806050217721857","timestamp":1588934073498,"message":"1588934073498560,test-aurora-master-instance,rdsadmin,localhost,3,4043,WRITE,mysql,rds_heartbeat2,"},{"id":"35434413910520633715885803047769579923953992942265565186","timestamp":1588934073506,"message":"1588934073506181,test-aurora-master-instance,rdsadmin,localhost,3,4043,QUERY,mysql,'INSERT INTO mysql.rds_heartbeat2(id, value) values (1,1588934073437) ON DUPLICATE KEY UPDATE value = 1588934073437',0"},{"id":"35434413911702573211407926074270972992404356102082527235","timestamp":1588934073559,"message":"1588934073559559,test-aurora-master-instance,rdsadmin,localhost,3,4045,QUERY,mysql,'COMMIT',0"},{"id":"35434413912639204509746212246215473159855587285333704708","timestamp":1588934073601,"message":"1588934073601235,test-aurora-master-instance,rdsadmin,localhost,3,4046,QUERY,mysql,'SET @@sql_log_bin=on',0"}]}{"messageType":"DATA_MESSAGE","owner":"XXXXXXXXXXXX","logGroup":"/aws/rds/cluster/test-aurora-cluster/audit","logStream":"test-aurora-master-instance.audit.log.1.2020-05-08-10-22.0.2","subscriptionFilters":["Destination"],"logEvents":[{"id":"35434413923165156243452666368978116711989928010403282944","timestamp":1588934074073,"message":"1588934074073147,test-aurora-master-instance,rdsadmin,localhost,2,4047,QUERY,mysql,'set local oscar_local_only_replica_host_status=1',0"},{"id":"35434413923165156243452666368978116711989928010403282945","timestamp":1588934074073,"message":"1588934074073570,test-aurora-master-instance,rdsadmin,localhost,2,4048,QUERY,mysql,'SELECT durable_lsn, current_read_point, server_id, last_update_timestamp FROM information_schema.replica_host_status',0"},{"id":"35434413923165156243452666368978116711989928010403282946","timestamp":1588934074073,"message":"1588934074073734,test-aurora-master-instance,rdsadmin,localhost,2,4049,QUERY,mysql,'set local oscar_local_only_replica_host_status=0',0"}]}{"messageType":"DATA_MESSAGE","owner":"XXXXXXXXXXXX","logGroup":"/aws/rds/cluster/test-aurora-cluster/audit","logStream":"test-aurora-master-instance.audit.log.3.2020-05-08-10-22.0.3","subscriptionFilters":["Destination"],"logEvents":[{"id":"35434413911234257562238782988377471568555818593442332672","timestamp":1588934073538,"message":"1588934073538633,test-aurora-master-instance,rdsadmin,localhost,3,4044,QUERY,mysql,'select @@session.tx_read_only',0"}]}{"messageType":"DATA_MESSAGE","owner":"XXXXXXXXXXXX","logGroup":"/aws/rds/cluster/test-aurora-cluster/audit","logStream":"test-aurora-master-instance.audit.log.1.2020-05-08-10-22.0.2","subscriptionFilters":["Destination"],"logEvents":[{"id":"35434413954386199521395538769657280867118860687645081600","timestamp":1588934075473,"message":"1588934075473820,test-aurora-master-instance,rdsadmin,localhost,3,4051,QUERY,mysql,'select @@session.tx_read_only',0"},{"id":"35434413954899116660961743101912602387389773002282631169","timestamp":1588934075496,"message":"1588934075496618,test-aurora-master-instance,rdsadmin,localhost,3,4052,WRITE,mysql,rds_heartbeat2,"},{"id":"35434413955545838271719131173017138217296575485956063234","timestamp":1588934075525,"message":"1588934075525979,test-aurora-master-instance,rdsadmin,localhost,3,4053,QUERY,mysql,'select @@session.tx_read_only',0"},{"id":"35434413957575206084785417878896888580107576383000281091","timestamp":1588934075616,"message":"1588934075616550,test-aurora-master-instance,rdsadmin,localhost,3,4055,QUERY,mysql,'SET @@sql_log_bin=on',0"}]}{"messageType":"DATA_MESSAGE","owner":"XXXXXXXXXXXX","logGroup":"/aws/rds/cluster/test-aurora-cluster/audit","logStream":"test-aurora-master-instance.audit.log.3.2020-05-08-10-22.0.3","subscriptionFilters":["Destination"],"logEvents":[{"id":"35434413952847448102696925772813648274814299131045937152","timestamp":1588934075404,"message":"1588934075404710,test-aurora-master-instance,rdsadmin,localhost,3,4050,QUERY,mysql,'SET @@sql_log_bin=off',0"}]}{"messageType":"DATA_MESSAGE","owner":"XXXXXXXXXXXX","logGroup":"/aws/rds/cluster/test-aurora-cluster/audit","logStream":"test-aurora-master-instance.audit.log.0.2020-05-08-10-22.0.0","subscriptionFilters":["Destination"],"logEvents":[{"id":"35434413955322830819733824941566666056484011352517705728","timestamp":1588934075515,"message":"1588934075515960,test-aurora-master-instance,rdsadmin,localhost,3,4052,QUERY,mysql,'INSERT INTO mysql.rds_heartbeat2(id, value) values (1,1588934075404) ON DUPLICATE KEY UPDATE value = 1588934075404',0"},{"id":"35434413956638574786447131706917273434570264681370550273","timestamp":1588934075574,"message":"1588934075574390,test-aurora-master-instance,rdsadmin,localhost,3,4054,QUERY,mysql,'COMMIT',0"},{"id":"35434413967119925029756524583439061022714994589181345794","timestamp":1588934076044,"message":"1588934076044035,test-aurora-master-instance,rdsadmin,localhost,2,4056,QUERY,mysql,'set local oscar_local_only_replica_host_status=1',0"},{"id":"35434413967119925029756524583439061022714994589181345795","timestamp":1588934076044,"message":"1588934076044467,test-aurora-master-instance,rdsadmin,localhost,2,4057,QUERY,mysql,'SELECT durable_lsn, current_read_point, server_id, last_update_timestamp FROM information_schema.replica_host_status',0"},{"id":"35434413967119925029756524583439061022714994589181345796","timestamp":1588934076044,"message":"1588934076044723,test-aurora-master-instance,rdsadmin,localhost,2,4058,QUERY,mysql,'set local oscar_local_only_replica_host_status=0',0"}]}{"messageType":"DATA_MESSAGE","owner":"XXXXXXXXXXXX","logGroup":"/aws/rds/cluster/test-aurora-cluster/audit","logStream":"test-aurora-master-instance.audit.log.3.2020-05-08-10-22.0.1","subscriptionFilters":["Destination"],"logEvents":[{"id":"35434413997671945951743478289700377487773100142831861760","timestamp":1588934077414,"message":"1588934077414191,test-aurora-master-instance,rdsadmin,localhost,3,4059,QUERY,mysql,'SET @@sql_log_bin=off',0"}]}{"messageType":"DATA_MESSAGE","owner":"XXXXXXXXXXXX","logGroup":"/aws/rds/cluster/test-aurora-cluster/audit","logStream":"test-aurora-master-instance.audit.log.2.2020-05-08-10-22.0.3","subscriptionFilters":["Destination"],"logEvents":[{"id":"35434413997738848187339070159169375225665883996102787072","timestamp":1588934077417,"message":"1588934077417297,test-aurora-master-instance,rdsadmin,localhost,3,4060,QUERY,mysql,'select @@session.tx_read_only',0"},{"id":"35434413999433704822427397517926089814387159470557298689","timestamp":1588934077493,"message":"1588934077493631,test-aurora-master-instance,rdsadmin,localhost,3,4061,WRITE,mysql,rds_heartbeat2,"},{"id":"35434413999679013019611234372482982715386291447123083266","timestamp":1588934077504,"message":"1588934077504455,test-aurora-master-instance,rdsadmin,localhost,3,4061,QUERY,mysql,'INSERT INTO mysql.rds_heartbeat2(id, value) values (1,1588934077414) ON DUPLICATE KEY UPDATE value = 1588934077414',0"},{"id":"35434413999924321216795071227039875616385423423688867843","timestamp":1588934077515,"message":"1588934077515718,test-aurora-master-instance,rdsadmin,localhost,3,4062,QUERY,mysql,'select @@session.tx_read_only',0"},{"id":"35434413999991223452390663096464482771203368508206809092","timestamp":1588934077518,"message":"1588934077518281,test-aurora-master-instance,rdsadmin,localhost,3,4063,QUERY,mysql,'COMMIT',0"},{"id":"35434414001842185303868704817211947387833182513203183621","timestamp":1588934077601,"message":"1588934077601397,test-aurora-master-instance,rdsadmin,localhost,3,4064,QUERY,mysql,'SET @@sql_log_bin=on',0"},{"id":"35434414011342302758442750275506163371981384514750840838","timestamp":1588934078027,"message":"1588934078027886,test-aurora-master-instance,rdsadmin,localhost,2,4065,QUERY,mysql,'set local oscar_local_only_replica_host_status=1',0"},{"id":"35434414011364603503641280898647699090254032876256821255","timestamp":1588934078028,"message":"1588934078028386,test-aurora-master-instance,rdsadmin,localhost,2,4066,QUERY,mysql,'SELECT durable_lsn, current_read_point, server_id, last_update_timestamp FROM information_schema.replica_host_status',0"},{"id":"35434414011386904248839811521789234808526681237762801672","timestamp":1588934078029,"message":"1588934078029162,test-aurora-master-instance,rdsadmin,localhost,2,4067,QUERY,mysql,'set local oscar_local_only_replica_host_status=0',0"}]}{"messageType":"DATA_MESSAGE","owner":"XXXXXXXXXXXX","logGroup":"/aws/rds/cluster/test-aurora-cluster/audit","logStream":"test-aurora-master-instance.audit.log.2.2020-05-08-10-22.0.0","subscriptionFilters":["Destination"],"logEvents":[{"id":"35434414043611481060716561963684130989092577376486490112","timestamp":1588934079474,"message":"1588934079474589,test-aurora-master-instance,rdsadmin,localhost,3,4068,QUERY,mysql,'SET @@sql_log_bin=off',0"},{"id":"35434414044369706397466603150496345410362621667689824257","timestamp":1588934079508,"message":"1588934079508937,test-aurora-master-instance,rdsadmin,localhost,3,4069,QUERY,mysql,'select @@session.tx_read_only',0"},{"id":"35434414045863856325768154900979238534630061888590512130","timestamp":1588934079575,"message":"1588934079575938,test-aurora-master-instance,rdsadmin,localhost,3,4070,WRITE,mysql,rds_heartbeat2,"},{"id":"35434414046287570484540236740668417181810380757204140035","timestamp":1588934079594,"message":"1588934079594638,test-aurora-master-instance,rdsadmin,localhost,3,4070,QUERY,mysql,'INSERT INTO mysql.rds_heartbeat2(id, value) values (1,1588934079474) ON DUPLICATE KEY UPDATE value = 1588934079474',0"},{"id":"35434414046622081662518196087791452955900106179793846276","timestamp":1588934079609,"message":"1588934079609699,test-aurora-master-instance,rdsadmin,localhost,3,4071,QUERY,mysql,'select @@session.tx_read_only',0"},{"id":"35434414048160833081216809084557417516712843123706494981","timestamp":1588934079678,"message":"1588934079678335,test-aurora-master-instance,rdsadmin,localhost,3,4072,QUERY,mysql,'COMMIT',0"},{"id":"35434414048539945749591829677963524727347865269308162054","timestamp":1588934079695,"message":"1588934079695911,test-aurora-master-instance,rdsadmin,localhost,3,4073,QUERY,mysql,'SET @@sql_log_bin=on',0"},{"id":"35434414057014228925033466471747097670954242641580720135","timestamp":1588934080075,"message":"1588934080075470,test-aurora-master-instance,rdsadmin,localhost,2,4074,QUERY,mysql,'set local oscar_local_only_replica_host_status=1',0"},{"id":"35434414057036529670231997094888633389226891003086700552","timestamp":1588934080076,"message":"1588934080076218,test-aurora-master-instance,rdsadmin,localhost,2,4075,QUERY,mysql,'SELECT durable_lsn, current_read_point, server_id, last_update_timestamp FROM information_schema.replica_host_status',0"},{"id":"35434414057036529670231997094888633389226891003086700553","timestamp":1588934080076,"message":"1588934080076670,test-aurora-master-instance,rdsadmin,localhost,2,4076,QUERY,mysql,'set local oscar_local_only_replica_host_status=0',0"}]}{"messageType":"DATA_MESSAGE","owner":"XXXXXXXXXXXX","logGroup":"/aws/rds/cluster/test-aurora-cluster/audit","logStream":"test-aurora-master-instance.audit.log.2.2020-05-08-10-22.0.2","subscriptionFilters":["Destination"],"logEvents":[{"id":"35434414087410144630630705816129836123378868702045470720","timestamp":1588934081438,"message":"1588934081438724,test-aurora-master-instance,rdsadmin,localhost,3,4078,QUERY,mysql,'select @@session.tx_read_only',0"},{"id":"35434414087811558044204257032677479052286539209153118209","timestamp":1588934081456,"message":"1588934081456773,test-aurora-master-instance,rdsadmin,localhost,3,4079,WRITE,mysql,rds_heartbeat2,"},{"id":"35434414088056866241388093887234371953285671185718902786","timestamp":1588934081467,"message":"1588934081467464,test-aurora-master-instance,rdsadmin,localhost,3,4079,QUERY,mysql,'INSERT INTO mysql.rds_heartbeat2(id, value) values (1,1588934081405) ON DUPLICATE KEY UPDATE value = 1588934081405',0"},{"id":"35434414089216504991711686290594229303463385984029884419","timestamp":1588934081519,"message":"1588934081519522,test-aurora-master-instance,rdsadmin,localhost,3,4081,QUERY,mysql,'COMMIT',0"}]}{"messageType":"DATA_MESSAGE","owner":"XXXXXXXXXXXX","logGroup":"/aws/rds/cluster/test-aurora-cluster/audit","logStream":"test-aurora-master-instance.audit.log.1.2020-05-08-10-22.0.2","subscriptionFilters":["Destination"],"logEvents":[{"id":"35434414101125102927727039048122579029527613545121316864","timestamp":1588934082053,"message":"1588934082053443,test-aurora-master-instance,rdsadmin,localhost,2,4083,QUERY,mysql,'set local oscar_local_only_replica_host_status=1',0"},{"id":"35434414101214305908521161540688721902618206991145238529","timestamp":1588934082057,"message":"1588934082057489,test-aurora-master-instance,rdsadmin,localhost,2,4084,QUERY,mysql,'SELECT durable_lsn, current_read_point, server_id, last_update_timestamp FROM information_schema.replica_host_status',0"},{"id":"35434414101236606653719692163830257620890855352651218946","timestamp":1588934082058,"message":"1588934082058026,test-aurora-master-instance,rdsadmin,localhost,2,4085,QUERY,mysql,'set local oscar_local_only_replica_host_status=0',0"}]}{"messageType":"DATA_MESSAGE","owner":"XXXXXXXXXXXX","logGroup":"/aws/rds/cluster/test-aurora-cluster/audit","logStream":"test-aurora-master-instance.audit.log.3.2020-05-08-10-22.0.3","subscriptionFilters":["Destination"],"logEvents":[{"id":"35434414086651919293880664629352654641583554691581739008","timestamp":1588934081404,"message":"1588934081404490,test-aurora-master-instance,rdsadmin,localhost,3,4077,QUERY,mysql,'SET @@sql_log_bin=off',0"},{"id":"35434414088703587852145481958373940722667203950131937281","timestamp":1588934081496,"message":"1588934081496686,test-aurora-master-instance,rdsadmin,localhost,3,4080,QUERY,mysql,'select @@session.tx_read_only',0"},{"id":"35434414089662519895682298753459976608391083494889095170","timestamp":1588934081539,"message":"1588934081539576,test-aurora-master-instance,rdsadmin,localhost,3,4082,QUERY,mysql,'SET @@sql_log_bin=on',0"}]}{"messageType":"DATA_MESSAGE","owner":"XXXXXXXXXXXX","logGroup":"/aws/rds/cluster/test-aurora-cluster/audit","logStream":"test-aurora-master-instance.audit.log.2.2020-05-08-10-22.0.3","subscriptionFilters":["Destination"],"logEvents":[{"id":"35434414133795694643574401952969763289862312102052364288","timestamp":1588934083518,"message":"1588934083518964,test-aurora-master-instance,rdsadmin,localhost,3,4089,QUERY,mysql,'select @@session.tx_read_only',0"}]}{"messageType":"DATA_MESSAGE","owner":"XXXXXXXXXXXX","logGroup":"/aws/rds/cluster/test-aurora-cluster/audit","logStream":"test-aurora-master-instance.audit.log.1.2020-05-08-10-22.0.2","subscriptionFilters":["Destination"],"logEvents":[{"id":"35434414133907198369567055068596597081082003104304136192","timestamp":1588934083523,"message":"1588934083523110,test-aurora-master-instance,rdsadmin,localhost,3,4090,QUERY,mysql,'COMMIT',0"},{"id":"35434414135156040100684769964522597304350311348639039489","timestamp":1588934083579,"message":"1588934083579274,test-aurora-master-instance,rdsadmin,localhost,3,4091,QUERY,mysql,'SET @@sql_log_bin=on',0"},{"id":"35434414145949600776773591565025884948312118317533560834","timestamp":1588934084063,"message":"1588934084063423,test-aurora-master-instance,rdsadmin,localhost,2,4093,QUERY,mysql,'SELECT durable_lsn, current_read_point, server_id, last_update_timestamp FROM information_schema.replica_host_status',0"},{"id":"35434414145949600776773591565025884948312118317533560835","timestamp":1588934084063,"message":"1588934084063981,test-aurora-master-instance,rdsadmin,localhost,2,4094,QUERY,mysql,'set local oscar_local_only_replica_host_status=0',0"}]}{"messageType":"DATA_MESSAGE","owner":"XXXXXXXXXXXX","logGroup":"/aws/rds/cluster/test-aurora-cluster/audit","logStream":"test-aurora-master-instance.audit.log.0.2020-05-08-10-22.0.3","subscriptionFilters":["Destination"],"logEvents":[{"id":"35434414131253409690941910914791683802429241966118371328","timestamp":1588934083404,"message":"1588934083404346,test-aurora-master-instance,rdsadmin,localhost,3,4086,QUERY,mysql,'SET @@sql_log_bin=off',0"},{"id":"35434414132502251422059625810717684025697550210453274625","timestamp":1588934083460,"message":"1588934083460657,test-aurora-master-instance,rdsadmin,localhost,3,4088,WRITE,mysql,rds_heartbeat2,"},{"id":"35434414132591454402853748303283826898788143656477196290","timestamp":1588934083464,"message":"1588934083464385,test-aurora-master-instance,rdsadmin,localhost,3,4088,QUERY,mysql,'INSERT INTO mysql.rds_heartbeat2(id, value) values (1,1588934083404) ON DUPLICATE KEY UPDATE value = 1588934083404',0"},{"id":"35434414145927300031575060941922186425831863837053485059","timestamp":1588934084062,"message":"1588934084062938,test-aurora-master-instance,rdsadmin,localhost,2,4092,QUERY,mysql,'set local oscar_local_only_replica_host_status=1',0"}]}{"messageType":"DATA_MESSAGE","owner":"XXXXXXXXXXXX","logGroup":"/aws/rds/cluster/test-aurora-cluster/audit","logStream":"test-aurora-master-instance.audit.log.3.2020-05-08-10-22.0.2","subscriptionFilters":["Destination"],"logEvents":[{"id":"35434414131543319378522809015712446520119995310445035520","timestamp":1588934083417,"message":"1588934083417596,test-aurora-master-instance,rdsadmin,localhost,3,4087,QUERY,mysql,'select @@session.tx_read_only',0"}]}{"messageType":"DATA_MESSAGE","owner":"XXXXXXXXXXXX","logGroup":"/aws/rds/cluster/test-aurora-cluster/audit","logStream":"test-aurora-master-instance.audit.log.3.2020-05-08-10-22.0.2","subscriptionFilters":["Destination"],"logEvents":[{"id":"35434414168629458643679235301157643400492767427251470336","timestamp":1588934085080,"message":"1588934085080080,test-aurora-master-instance,rdsadmin,localhost,43,0,CONNECT,,,0"},{"id":"35434414172420585327429441235218715506842988883268141057","timestamp":1588934085250,"message":"1588934085250720,test-aurora-master-instance,rdsadmin,localhost,43,4102,QUERY,,'/* mysql-connector-java-5.1.33 ( Revision: [email protected] ) */SHOW VARIABLES WHERE Variable_name =\\'language\\' OR Variable_name = \\'net_write_timeout\\' OR Variable_name = \\'interactive_timeout\\' OR Variable_name = \\'wait_timeout\\' OR Variable_name = \\'character_set_client\\' OR Variable_name = \\'character_set_connection\\' OR Variable_name = \\'character_set\\' OR Variable_name = \\'character_set_server\\' OR Variable_name = \\'tx_isolation\\' OR Variable_name = \\'transaction_isolation\\' OR Variable_name = \\'character_set_results\\' OR Variable_name = \\'timezone\\' OR Variable_name = \\'time_zone\\' OR Variable_name = \\'system_time_zone\\' OR Variable_name = \\'lower_case_table_names\\' OR Variable_name = \\'max_allowed_packet\\' OR Variable_name = \\'net_buffer_length\\' OR Variable_name = \\'sql_mode\\' OR Variable_name = \\'query_cache_type\\' OR Variable_name = \\'query_cache_size\\' OR Variable_name = \\'license\\' OR Variable_name = \\'init_connect\\'',0"}]}{"messageType":"DATA_MESSAGE","owner":"XXXXXXXXXXXX","logGroup":"/aws/rds/cluster/test-aurora-cluster/audit","logStream":"test-aurora-master-instance.audit.log.2.2020-05-08-10-22.0.2","subscriptionFilters":["Destination"],"logEvents":[{"id":"35434414169699894413208705212013808182245207078061539328","timestamp":1588934085128,"message":"1588934085128524,test-aurora-master-instance,rdsadmin,localhost,2,4096,QUERY,mysql,'set local oscar_local_only_replica_host_status=1',0"},{"id":"35434414169699894413208705212013808182245207078061539329","timestamp":1588934085128,"message":"1588934085128818,test-aurora-master-instance,rdsadmin,localhost,43,4095,QUERY,,'SET SESSION wait_timeout=28800',0"},{"id":"35434414169722195158407235835155343900517855439567519746","timestamp":1588934085129,"message":"1588934085129160,test-aurora-master-instance,rdsadmin,localhost,2,4097,QUERY,mysql,'SELECT durable_lsn, current_read_point, server_id, last_update_timestamp FROM information_schema.replica_host_status',0"},{"id":"35434414169722195158407235835155343900517855439567519747","timestamp":1588934085129,"message":"1588934085129903,test-aurora-master-instance,rdsadmin,localhost,2,4098,QUERY,mysql,'set local oscar_local_only_replica_host_status=0',0"},{"id":"35434414172710495015010339336121130149052735881368895492","timestamp":1588934085263,"message":"1588934085263930,test-aurora-master-instance,rdsadmin,localhost,43,4103,QUERY,,'/* mysql-connector-java-5.1.33 ( Revision: [email protected] ) */SELECT @@session.auto_increment_increment',0"},{"id":"35434414173111908428583890552668773077960406388476542981","timestamp":1588934085281,"message":"1588934085281906,test-aurora-master-instance,rdsadmin,localhost,43,4104,QUERY,,'SET NAMES utf8',0"},{"id":"35434414174672960592481034172576273357045791693895172102","timestamp":1588934085351,"message":"1588934085351694,test-aurora-master-instance,rdsadmin,localhost,43,4106,QUERY,,'SET autocommit=1',0"},{"id":"35434414175029772515657524142840844849408165477990858759","timestamp":1588934085367,"message":"1588934085367719,test-aurora-master-instance,rdsadmin,localhost,43,4107,QUERY,,'SET sql_mode=\\'STRICT_TRANS_TABLES\\'',0"},{"id":"35434414175364283693635483489963880623497890900580565000","timestamp":1588934085382,"message":"1588934085382328,test-aurora-master-instance,rdsadmin,localhost,43,4108,QUERY,,'SHOW WARNINGS',0"}]}{"messageType":"DATA_MESSAGE","owner":"XXXXXXXXXXXX","logGroup":"/aws/rds/cluster/test-aurora-cluster/audit","logStream":"test-aurora-master-instance.audit.log.1.2020-05-08-10-22.0.1","subscriptionFilters":["Destination"],"logEvents":[{"id":"35434414170881833908730828238596081634708167061384527872","timestamp":1588934085181,"message":"1588934085181250,test-aurora-master-instance,rdsadmin,localhost,43,4100,QUERY,,'SET SESSION sql_mode=0',0"},{"id":"35434414171974570423458828772531331830067936775177568257","timestamp":1588934085230,"message":"1588934085230257,test-aurora-master-instance,rdsadmin,localhost,43,4101,QUERY,,'SET SESSION autocommit=1',0"}]}{"messageType":"DATA_MESSAGE","owner":"XXXXXXXXXXXX","logGroup":"/aws/rds/cluster/test-aurora-cluster/audit","logStream":"test-aurora-master-instance.audit.log.0.2020-05-08-10-22.0.3","subscriptionFilters":["Destination"],"logEvents":[{"id":"35434414189636760620695082301736303058535348151994941440","timestamp":1588934086022,"message":"1588934086022705,test-aurora-master-instance,rdsadmin,localhost,2,4117,QUERY,mysql,'SELECT durable_lsn, current_read_point, server_id, last_update_timestamp FROM information_schema.replica_host_status',0"}]}{"messageType":"DATA_MESSAGE","owner":"XXXXXXXXXXXX","logGroup":"/aws/rds/cluster/test-aurora-cluster/audit","logStream":"test-aurora-master-instance.audit.log.0.2020-05-08-10-22.0.0","subscriptionFilters":["Destination"],"logEvents":[{"id":"35434414170145909317179317674881982504443169350927450112","timestamp":1588934085148,"message":"1588934085148680,test-aurora-master-instance,rdsadmin,localhost,43,4099,QUERY,,'SET SESSION time_zone=UTC',0"},{"id":"35434414174226945688510421709783018948337819506521866241","timestamp":1588934085331,"message":"1588934085331446,test-aurora-master-instance,rdsadmin,localhost,43,4105,QUERY,,'SET character_set_results = NULL',0"}]}{"messageType":"DATA_MESSAGE","owner":"XXXXXXXXXXXX","logGroup":"/aws/rds/cluster/test-aurora-cluster/audit","logStream":"test-aurora-master-instance.audit.log.3.2020-05-08-10-22.0.2","subscriptionFilters":["Destination"],"logEvents":[{"id":"35434414176925335857532627111016577689139404565827944448","timestamp":1588934085452,"message":"1588934085452874,test-aurora-master-instance,rdsadmin,localhost,3,4111,QUERY,mysql,'select @@session.tx_read_only',0"},{"id":"35434414177728162684679729544111863546954745580043239425","timestamp":1588934085488,"message":"1588934085488876,test-aurora-master-instance,rdsadmin,localhost,3,4113,QUERY,mysql,'select @@session.tx_read_only',0"},{"id":"35434414179177711122584220048311685234676889077931966466","timestamp":1588934085553,"message":"1588934085553926,test-aurora-master-instance,rdsadmin,localhost,3,4115,QUERY,mysql,'SET @@sql_log_bin=on',0"},{"id":"35434414189636760620695082301691937104548970624236781571","timestamp":1588934086022,"message":"1588934086022238,test-aurora-master-instance,rdsadmin,localhost,2,4116,QUERY,mysql,'set local oscar_local_only_replica_host_status=1',0"},{"id":"35434414189659061365893612924833472822821618985742761988","timestamp":1588934086023,"message":"1588934086023156,test-aurora-master-instance,rdsadmin,localhost,2,4118,QUERY,mysql,'set local oscar_local_only_replica_host_status=0',0"}]}{"messageType":"DATA_MESSAGE","owner":"XXXXXXXXXXXX","logGroup":"/aws/rds/cluster/test-aurora-cluster/audit","logStream":"test-aurora-master-instance.audit.log.2.2020-05-08-10-22.0.1","subscriptionFilters":["Destination"],"logEvents":[{"id":"35434414176457020208363484025126095783125125437952360448","timestamp":1588934085431,"message":"1588934085431938,test-aurora-master-instance,rdsadmin,localhost,43,4109,QUERY,,'SELECT @@aurora_version',0"},{"id":"35434414176479320953562014648267631501397773799458340865","timestamp":1588934085432,"message":"1588934085432364,test-aurora-master-instance,rdsadmin,localhost,3,4110,QUERY,mysql,'SET @@sql_log_bin=off',0"},{"id":"35434414177282147780709117081362917359213114813673635842","timestamp":1588934085468,"message":"1588934085468912,test-aurora-master-instance,rdsadmin,localhost,3,4112,WRITE,mysql,rds_heartbeat2,"},{"id":"35434414177549756723091484559061345978484895151745400835","timestamp":1588934085480,"message":"1588934085480504,test-aurora-master-instance,rdsadmin,localhost,3,4112,QUERY,mysql,'INSERT INTO mysql.rds_heartbeat2(id, value) values (1,1588934085432) ON DUPLICATE KEY UPDATE value = 1588934085432',0"},{"id":"35434414178753996963812138208704274765207906673068343300","timestamp":1588934085534,"message":"1588934085534425,test-aurora-master-instance,rdsadmin,localhost,3,4114,QUERY,mysql,'COMMIT',0"}]}{"messageType":"DATA_MESSAGE","owner":"XXXXXXXXXXXX","logGroup":"/aws/rds/cluster/test-aurora-cluster/audit","logStream":"test-aurora-master-instance.audit.log.2.2020-05-08-10-22.0.2","subscriptionFilters":["Destination"],"logEvents":[{"id":"35434414223979908226432241942128825191250155059812761600","timestamp":1588934087562,"message":"1588934087562959,test-aurora-master-instance,rdsadmin,localhost,3,4124,QUERY,mysql,'SET @@sql_log_bin=on',0"}]}{"messageType":"DATA_MESSAGE","owner":"XXXXXXXXXXXX","logGroup":"/aws/rds/cluster/test-aurora-cluster/audit","logStream":"test-aurora-master-instance.audit.log.0.2020-05-08-10-22.0.1","subscriptionFilters":["Destination"],"logEvents":[{"id":"35434414220746300172645301586641563200296746566464241664","timestamp":1588934087417,"message":"1588934087417470,test-aurora-master-instance,rdsadmin,localhost,3,4119,QUERY,mysql,'SET @@sql_log_bin=off',0"},{"id":"35434414222218149355748322713982920606291538425858949121","timestamp":1588934087483,"message":"1588934087483379,test-aurora-master-instance,rdsadmin,localhost,3,4121,WRITE,mysql,rds_heartbeat2,"},{"id":"35434414223377788106071915117342777956469253224169930754","timestamp":1588934087535,"message":"1588934087535078,test-aurora-master-instance,rdsadmin,localhost,3,4122,QUERY,mysql,'select @@session.tx_read_only',0"},{"id":"35434414223400088851270445740484313674741901585675911171","timestamp":1588934087536,"message":"1588934087536123,test-aurora-master-instance,rdsadmin,localhost,3,4123,QUERY,mysql,'COMMIT',0"}]}{"messageType":"DATA_MESSAGE","owner":"XXXXXXXXXXXX","logGroup":"/aws/rds/cluster/test-aurora-cluster/audit","logStream":"test-aurora-master-instance.audit.log.3.2020-05-08-10-22.0.3","subscriptionFilters":["Destination"],"logEvents":[{"id":"35434414222329653081740975829616640168821148902769229824","timestamp":1588934087488,"message":"1588934087488222,test-aurora-master-instance,rdsadmin,localhost,3,4121,QUERY,mysql,'INSERT INTO mysql.rds_heartbeat2(id, value) values (1,1588934087417) ON DUPLICATE KEY UPDATE value = 1588934087417',0"},{"id":"35434414235509393494072574106264249667956330552803655681","timestamp":1588934088079,"message":"1588934088079694,test-aurora-master-instance,rdsadmin,localhost,2,4125,QUERY,mysql,'set local oscar_local_only_replica_host_status=1',0"},{"id":"35434414235531694239271104729405785386228978914309636098","timestamp":1588934088080,"message":"1588934088080209,test-aurora-master-instance,rdsadmin,localhost,2,4126,QUERY,mysql,'SELECT durable_lsn, current_read_point, server_id, last_update_timestamp FROM information_schema.replica_host_status',0"},{"id":"35434414235531694239271104729405785386228978914309636099","timestamp":1588934088080,"message":"1588934088080604,test-aurora-master-instance,rdsadmin,localhost,2,4127,QUERY,mysql,'set local oscar_local_only_replica_host_status=0',0"}]}{"messageType":"DATA_MESSAGE","owner":"XXXXXXXXXXXX","logGroup":"/aws/rds/cluster/test-aurora-cluster/audit","logStream":"test-aurora-master-instance.audit.log.1.2020-05-08-10-22.0.3","subscriptionFilters":["Destination"],"logEvents":[{"id":"35434414221727532961380649004905234281702961398451011584","timestamp":1588934087461,"message":"1588934087461583,test-aurora-master-instance,rdsadmin,localhost,3,4120,QUERY,mysql,'select @@session.tx_read_only',0"}]}{"messageType":"DATA_MESSAGE","owner":"XXXXXXXXXXXX","logGroup":"/aws/rds/cluster/test-aurora-cluster/audit","logStream":"test-aurora-master-instance.audit.log.0.2020-05-08-10-22.0.3","subscriptionFilters":["Destination"],"logEvents":[{"id":"35434414266284421868044834044084803122597585808883318784","timestamp":1588934089459,"message":"1588934089459430,test-aurora-master-instance,rdsadmin,localhost,3,4130,WRITE,mysql,rds_heartbeat2,"},{"id":"35434414267890075522339038910275374838228267837313908737","timestamp":1588934089531,"message":"1588934089531938,test-aurora-master-instance,rdsadmin,localhost,3,4133,QUERY,mysql,'SET @@sql_log_bin=on',0"}]}{"messageType":"DATA_MESSAGE","owner":"XXXXXXXXXXXX","logGroup":"/aws/rds/cluster/test-aurora-cluster/audit","logStream":"test-aurora-master-instance.audit.log.1.2020-05-08-10-22.0.0","subscriptionFilters":["Destination"],"logEvents":[{"id":"35434414266529730065228670898680592637099283915319279616","timestamp":1588934089470,"message":"1588934089470062,test-aurora-master-instance,rdsadmin,localhost,3,4130,QUERY,mysql,'INSERT INTO mysql.rds_heartbeat2(id, value) values (1,1588934089425) ON DUPLICATE KEY UPDATE value = 1588934089425',0"},{"id":"35434414267778571796346385794606592860367592159654182913","timestamp":1588934089526,"message":"1588934089526514,test-aurora-master-instance,rdsadmin,localhost,3,4132,QUERY,mysql,'COMMIT',0"}]}{"messageType":"DATA_MESSAGE","owner":"XXXXXXXXXXXX","logGroup":"/aws/rds/cluster/test-aurora-cluster/audit","logStream":"test-aurora-master-instance.audit.log.2.2020-05-08-10-22.0.2","subscriptionFilters":["Destination"],"logEvents":[{"id":"35434414265526196531294792857238923144470403114083155968","timestamp":1588934089425,"message":"1588934089425321,test-aurora-master-instance,rdsadmin,localhost,3,4128,QUERY,mysql,'SET @@sql_log_bin=off',0"},{"id":"35434414267555564344361079563118673507281404011127373825","timestamp":1588934089516,"message":"1588934089516651,test-aurora-master-instance,rdsadmin,localhost,3,4131,QUERY,mysql,'select @@session.tx_read_only',0"},{"id":"35434414278973545886008758611584961262877365102189346818","timestamp":1588934090028,"message":"1588934090028096,test-aurora-master-instance,rdsadmin,localhost,2,4134,QUERY,mysql,'set local oscar_local_only_replica_host_status=1',0"},{"id":"35434414278973545886008758611584961262877365102189346819","timestamp":1588934090028,"message":"1588934090028767,test-aurora-master-instance,rdsadmin,localhost,2,4135,QUERY,mysql,'SELECT durable_lsn, current_read_point, server_id, last_update_timestamp FROM information_schema.replica_host_status',0"},{"id":"35434414278995846631207289234726496981150013463695327236","timestamp":1588934090029,"message":"1588934090029113,test-aurora-master-instance,rdsadmin,localhost,2,4136,QUERY,mysql,'set local oscar_local_only_replica_host_status=0',0"}]}{"messageType":"DATA_MESSAGE","owner":"XXXXXXXXXXXX","logGroup":"/aws/rds/cluster/test-aurora-cluster/audit","logStream":"test-aurora-master-instance.audit.log.3.2020-05-08-10-22.0.2","subscriptionFilters":["Destination"],"logEvents":[{"id":"35434414265660001002485976596045713386919836301960740864","timestamp":1588934089431,"message":"1588934089431252,test-aurora-master-instance,rdsadmin,localhost,3,4129,QUERY,mysql,'select @@session.tx_read_only',0"}]}% sakamaki.kazuyoshi@HL
このままでは、Athenaで確認を行えないので、Kinesis Data Firehoseを介すところでLambda Functionにてデータの変換を行います。こちらについては、過去のBlack BeltのQAに取り上げられていました。
Q8. Amazon Kinesis Firehose を利用してCloudWatch LogsをS3に転送してそれをAthena で分析したいのですが、Kinesis Firehoseを通すと{json}{json}のように１行に複数のJSONオブジェクトが保存されるようです。このデータを効率的にAthenaで分析するにはどういった方法がありますか？？
A8. Amazon Kinesis FirehoseにはData TransformationをAWS Lambdaで行う機能がございますので，こちらを使って所望の形式に変換すると良いです.
Kinesis Data Firehose設定
配信ストリームの設定は以下となります。
配信ストリーム設定
$ aws firehose describe-delivery-stream \
--delivery-stream-name delivery-stream-for-athena
{
"DeliveryStreamDescription": {
"DeliveryStreamName": "delivery-stream-for-athena",
"DeliveryStreamARN": "arn:aws:firehose:ap-northeast-1:XXXXXXXXXXXX:deliverystream/delivery-stream-for-athena",
"DeliveryStreamStatus": "ACTIVE",
"DeliveryStreamEncryptionConfiguration": {
"Status": "DISABLED"
},
"DeliveryStreamType": "DirectPut",
"VersionId": "3",
"CreateTimestamp": 1588932385.813,
"LastUpdateTimestamp": 1588932473.794,
"Destinations": [
{
"DestinationId": "destinationId-000000000001",
"S3DestinationDescription": {
"RoleARN": "arn:aws:iam::XXXXXXXXXXXX:role/TestFirehosetoS3Role",
"BucketARN": "arn:aws:s3:::cloudwatch-logs-for-athena-convert",
"Prefix": "test-aurora-cluster/audit/!{timestamp:'year='yyyy'/month='MM'/day='dd'/hour='HH}/",
"ErrorOutputPrefix": "error-test-aurora-cluster/audit/!{timestamp:'year='yyyy'/month='MM'/day='dd'/hour='HH}/!{firehose:error-output-type}",
"BufferingHints": {
"SizeInMBs": 1,
"IntervalInSeconds": 60
},
"CompressionFormat": "GZIP",
"EncryptionConfiguration": {
"NoEncryptionConfig": "NoEncryption"
},
"CloudWatchLoggingOptions": {
"Enabled": true,
"LogGroupName": "/aws/kinesisfirehose/delivery-stream-for-athena",
"LogStreamName": "S3Delivery"
}
},
"ExtendedS3DestinationDescription": {
"RoleARN": "arn:aws:iam::XXXXXXXXXXXX:role/TestFirehosetoS3Role",
"BucketARN": "arn:aws:s3:::cloudwatch-logs-for-athena-convert",
"Prefix": "test-aurora-cluster/audit/!{timestamp:'year='yyyy'/month='MM'/day='dd'/hour='HH}/",
"ErrorOutputPrefix": "error-test-aurora-cluster/audit/!{timestamp:'year='yyyy'/month='MM'/day='dd'/hour='HH}/!{firehose:error-output-type}",
"BufferingHints": {
"SizeInMBs": 1,
"IntervalInSeconds": 60
},
"CompressionFormat": "GZIP",
"EncryptionConfiguration": {
"NoEncryptionConfig": "NoEncryption"
},
"CloudWatchLoggingOptions": {
"Enabled": true,
"LogGroupName": "/aws/kinesisfirehose/delivery-stream-for-athena",
"LogStreamName": "S3Delivery"
},
"ProcessingConfiguration": {
"Enabled": true,
"Processors": [
{
"Type": "Lambda",
"Parameters": [
{
"ParameterName": "LambdaArn",
"ParameterValue": "arn:aws:lambda:ap-northeast-1:XXXXXXXXXXXX:function:kinesis-firehose-cloudwatch-logs-processor-python:$LATEST"
},
{
"ParameterName": "NumberOfRetries",
"ParameterValue": "3"
},
{
"ParameterName": "RoleArn",
"ParameterValue": "arn:aws:iam::XXXXXXXXXXXX:role/TestFirehosetoS3Role"
},
{
"ParameterName": "BufferSizeInMBs",
"ParameterValue": "1"
},
{
"ParameterName": "BufferIntervalInSeconds",
"ParameterValue": "60"
}
]
}
]
},
"S3BackupMode": "Enabled",
"S3BackupDescription": {
"RoleARN": "arn:aws:iam::XXXXXXXXXXXX:role/TestFirehosetoS3Role",
"BucketARN": "arn:aws:s3:::cloudwatch-logs-for-athena-backup",
"Prefix": "source_records/",
"ErrorOutputPrefix": "!{firehose:error-output-type}/",
"BufferingHints": {
"SizeInMBs": 5,
"IntervalInSeconds": 300
},
"CompressionFormat": "UNCOMPRESSED",
"EncryptionConfiguration": {
"NoEncryptionConfig": "NoEncryption"
},
"CloudWatchLoggingOptions": {
"Enabled": false
}
},
"DataFormatConversionConfiguration": {
"Enabled": false
}
}
}
],
"HasMoreDestinations": false
}
}
S3 Select時の設定から変更している箇所を中心に補足します。
ProcessingConfiguration
ログデータの変換にLambda Functionを指定しています。作り込みが発生する箇所です。そのまま利用できそうなBluePrintがあったので今回はそちらを利用しました。BluePrintのコードをこちらに添付しておきます。
kinesis-firehose-cloudwatch-logs-processor-python
"""
For processing data sent to Firehose by Cloudwatch Logs subscription filters.
Cloudwatch Logs sends to Firehose records that look like this:
{
"messageType": "DATA_MESSAGE",
"owner": "123456789012",
"logGroup": "log_group_name",
"logStream": "log_stream_name",
"subscriptionFilters": [
"subscription_filter_name"
],
"logEvents": [
{
"id": "01234567890123456789012345678901234567890123456789012345",
"timestamp": 1510109208016,
"message": "log message 1"
},
{
"id": "01234567890123456789012345678901234567890123456789012345",
"timestamp": 1510109208017,
"message": "log message 2"
}
...
]
}
The data is additionally compressed with GZIP.
The code below will:
1) Gunzip the data
2) Parse the json
3) Set the result to ProcessingFailed for any record whose messageType is not DATA_MESSAGE, thus redirecting them to the
processing error output. Such records do not contain any log events. You can modify the code to set the result to
Dropped instead to get rid of these records completely.
4) For records whose messageType is DATA_MESSAGE, extract the individual log events from the logEvents field, and pass
each one to the transformLogEvent method. You can modify the transformLogEvent method to perform custom
transformations on the log events.
5) Concatenate the result from (4) together and set the result as the data of the record returned to Firehose. Note that
this step will not add any delimiters. Delimiters should be appended by the logic within the transformLogEvent
method.
6) Any additional records which exceed 6MB will be re-ingested back into Firehose.
"""
import base64
import json
import gzip
import StringIO
import boto3
def transformLogEvent(log_event):
"""Transform each log event.
The default implementation below just extracts the message and appends a newline to it.
Args:
log_event (dict): The original log event. Structure is {"id": str, "timestamp": long, "message": str}
Returns:
str: The transformed log event.
"""
return log_event['message'] + '\n'
def processRecords(records):
for r in records:
data = base64.b64decode(r['data'])
striodata = StringIO.StringIO(data)
with gzip.GzipFile(fileobj=striodata, mode='r') as f:
data = json.loads(f.read())
recId = r['recordId']
"""
CONTROL_MESSAGE are sent by CWL to check if the subscription is reachable.
They do not contain actual data.
"""
if data['messageType'] == 'CONTROL_MESSAGE':
yield {
'result': 'Dropped',
'recordId': recId
}
elif data['messageType'] == 'DATA_MESSAGE':
data = ''.join([transformLogEvent(e) for e in data['logEvents']])
data = base64.b64encode(data)
yield {
'data': data,
'result': 'Ok',
'recordId': recId
}
else:
yield {
'result': 'ProcessingFailed',
'recordId': recId
}
def putRecordsToFirehoseStream(streamName, records, client, attemptsMade, maxAttempts):
failedRecords = []
codes = []
errMsg = ''
# if put_record_batch throws for whatever reason, response['xx'] will error out, adding a check for a valid
# response will prevent this
response = None
try:
response = client.put_record_batch(DeliveryStreamName=streamName, Records=records)
except Exception as e:
failedRecords = records
errMsg = str(e)
# if there are no failedRecords (put_record_batch succeeded), iterate over the response to gather results
if not failedRecords and response and response['FailedPutCount'] > 0:
for idx, res in enumerate(response['RequestResponses']):
# (if the result does not have a key 'ErrorCode' OR if it does and is empty) => we do not need to re-ingest
if 'ErrorCode' not in res or not res['ErrorCode']:
continue
codes.append(res['ErrorCode'])
failedRecords.append(records[idx])
errMsg = 'Individual error codes: ' + ','.join(codes)
if len(failedRecords) > 0:
if attemptsMade + 1 < maxAttempts:
print('Some records failed while calling PutRecordBatch to Firehose stream, retrying. %s' % (errMsg))
putRecordsToFirehoseStream(streamName, failedRecords, client, attemptsMade + 1, maxAttempts)
else:
raise RuntimeError('Could not put records after %s attempts. %s' % (str(maxAttempts), errMsg))
def putRecordsToKinesisStream(streamName, records, client, attemptsMade, maxAttempts):
failedRecords = []
codes = []
errMsg = ''
# if put_records throws for whatever reason, response['xx'] will error out, adding a check for a valid
# response will prevent this
response = None
try:
response = client.put_records(StreamName=streamName, Records=records)
except Exception as e:
failedRecords = records
errMsg = str(e)
# if there are no failedRecords (put_record_batch succeeded), iterate over the response to gather results
if not failedRecords and response and response['FailedRecordCount'] > 0:
for idx, res in enumerate(response['Records']):
# (if the result does not have a key 'ErrorCode' OR if it does and is empty) => we do not need to re-ingest
if 'ErrorCode' not in res or not res['ErrorCode']:
continue
codes.append(res['ErrorCode'])
failedRecords.append(records[idx])
errMsg = 'Individual error codes: ' + ','.join(codes)
if len(failedRecords) > 0:
if attemptsMade + 1 < maxAttempts:
print('Some records failed while calling PutRecords to Kinesis stream, retrying. %s' % (errMsg))
putRecordsToKinesisStream(streamName, failedRecords, client, attemptsMade + 1, maxAttempts)
else:
raise RuntimeError('Could not put records after %s attempts. %s' % (str(maxAttempts), errMsg))
def createReingestionRecord(isSas, originalRecord):
if isSas:
return {'data': base64.b64decode(originalRecord['data']), 'partitionKey': originalRecord['kinesisRecordMetadata']['partitionKey']}
else:
return {'data': base64.b64decode(originalRecord['data'])}
def getReingestionRecord(isSas, reIngestionRecord):
if isSas:
return {'Data': reIngestionRecord['data'], 'PartitionKey': reIngestionRecord['partitionKey']}
else:
return {'Data': reIngestionRecord['data']}
def handler(event, context):
isSas = 'sourceKinesisStreamArn' in event
streamARN = event['sourceKinesisStreamArn'] if isSas else event['deliveryStreamArn']
region = streamARN.split(':')[3]
streamName = streamARN.split('/')[1]
records = list(processRecords(event['records']))
projectedSize = 0
dataByRecordId = {rec['recordId']: createReingestionRecord(isSas, rec) for rec in event['records']}
putRecordBatches = []
recordsToReingest = []
totalRecordsToBeReingested = 0
for idx, rec in enumerate(records):
if rec['result'] != 'Ok':
continue
projectedSize += len(rec['data']) + len(rec['recordId'])
# 6000000 instead of 6291456 to leave ample headroom for the stuff we didn't account for
if projectedSize > 6000000:
totalRecordsToBeReingested += 1
recordsToReingest.append(
getReingestionRecord(isSas, dataByRecordId[rec['recordId']])
)
records[idx]['result'] = 'Dropped'
del(records[idx]['data'])
# split out the record batches into multiple groups, 500 records at max per group
if len(recordsToReingest) == 500:
putRecordBatches.append(recordsToReingest)
recordsToReingest = []
if len(recordsToReingest) > 0:
# add the last batch
putRecordBatches.append(recordsToReingest)
# iterate and call putRecordBatch for each group
recordsReingestedSoFar = 0
if len(putRecordBatches) > 0:
client = boto3.client('kinesis', region_name=region) if isSas else boto3.client('firehose', region_name=region)
for recordBatch in putRecordBatches:
if isSas:
putRecordsToKinesisStream(streamName, recordBatch, client, attemptsMade=0, maxAttempts=20)
else:
putRecordsToFirehoseStream(streamName, recordBatch, client, attemptsMade=0, maxAttempts=20)
recordsReingestedSoFar += len(recordBatch)
print('Reingested %d/%d records out of %d' % (recordsReingestedSoFar, totalRecordsToBeReingested, len(event['records'])))
else:
print('No records to be reingested')
return {"records": records}
なお、こちらのBluePrintはランタイムがPython 2.7のため、本番利用する際はコード変換などを検討してください。
なお、Kinesis Data FirehoseからのLambda Functionへ送信されるイベントは以下が参考になります。
S3DestinationDescription
出力先のバケットを変更し、呼び出しされるLambda Function（BluePrint）にて、圧縮データ（ログデータ）を展開しているので、S3へ出力する際にKinesis Data Firehoseにて圧縮しています。また、Athenaで解析がしやすいように、カスタムパーティションも設定しています。
CloudWatch Logsサブスクリプションフィルタ設定
サブスクリプションフィルタの設定は以下となります。
サブスクリプションフィルタ設定
$ aws logs describe-subscription-filters \
--log-group-name /aws/rds/cluster/test-aurora-cluster/audit
{
"subscriptionFilters": [
{
"filterName": "Destination",
"logGroupName": "/aws/rds/cluster/test-aurora-cluster/audit",
"filterPattern": "",
"destinationArn": "arn:aws:firehose:ap-northeast-1:XXXXXXXXXXXX:deliverystream/delivery-stream-for-athena",
"roleArn": "arn:aws:iam::XXXXXXXXXXXX:role/TestCWLtoKinesisFirehoseRole",
"distribution": "ByLogStream",
"creationTime": 1588935891719
}
]
}
CloudWatch Logロギング
後ほどログの確認が行いやすいよう、AuroraにアクセスしCloudWatch Logへロギングしました。
Auroraでのオペレーション
MySQL [(none)]> select now(); +---------------------+ | now() | +---------------------+ | 2020-05-08 20:06:09 | +---------------------+ 1 row in set (0.00 sec) MySQL [(none)]> select user(); +-----------------+ | user() | +-----------------+ | [email protected] | +-----------------+ 1 row in set (0.00 sec) MySQL [(none)]> select host,user,authentication_string from mysql.user; +-----------+-----------+-------------------------------------------+ | host | user | authentication_string | +-----------+-----------+-------------------------------------------+ | % | admin | *2470C0C06DEE42FD1618BB99005ADCA2EC9D1E19 | | % | test_user | *2470C0C06DEE42FD1618BB99005ADCA2EC9D1E19 | | % | testadmin | *2470C0C06DEE42FD1618BB99005ADCA2EC9D1E19 | | localhost | mysql.sys | *THISISNOTAVALIDPASSWORDTHATCANBEUSEDHERE | | localhost | rdsadmin | *8C0E64674BB3F606FF339142F68298EB1061E446 | +-----------+-----------+-------------------------------------------+ 5 rows in set (0.01 sec) MySQL [(none)]> select now(); +---------------------+ | now() | +---------------------+ | 2020-05-08 20:06:10 | +---------------------+ 1 row in set (0.00 sec)
しばらくすると、CloudWatch Logへのロギングが確認できました。
CloudWatch Logロギング確認
$ QUERY_ID=`aws logs start-query \
--log-group-name '/aws/rds/cluster/test-aurora-cluster/audit' \
--start-time 1588935950 \
--end-time 1588936010 \
--query-string \
'parse "*,*,*,*,*,*,*,*,*,*" as timestamp,serverhost,username,host,connectionid,queryid,operation,database,object,retcode
| filter username = "admin"
| sort timestamp asc' \
--output text`
$ aws logs get-query-results \
--query-id ${QUERY_ID} \
--output table
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
| GetQueryResults |
+-------------------------------------------------------------------------------------+--------------------------------------------------------------------------------------------------------+
| status | Complete |
+-------------------------------------------------------------------------------------+--------------------------------------------------------------------------------------------------------+
|| results ||
|+--------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+|
|| field | value ||
|+--------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+|
|| timestamp | 1588935964038921 ||
|| serverhost | test-aurora-master-instance ||
|| username | admin ||
|| host | 10.0.1.54 ||
|| connectionid| 137 ||
|| queryid | 0 ||
|| operation | CONNECT ||
|| database | ||
|| object | ||
|| retcode | 0 ||
|| @ptr | CnYKOwo3MzkyMTM4Nzg5NzY0Oi9hd3MvcmRzL2NsdXN0ZXIvdGVzdC1hdXJvcmEtY2x1c3Rlci9hdWRpdBAFEjcaGAIF55NVCwAAAAF4+3PpAAXrU9EwAAABsiABKIO42p+fLjCZ9eOfny44lAFA7NEBSKNOUMZKEAUYAQ== ||
|| timestamp | 1588935964039296 ||
|| serverhost | test-aurora-master-instance ||
|| username | admin ||
|| host | 10.0.1.54 ||
|| connectionid| 137 ||
|| queryid | 14821 ||
|| operation | QUERY ||
|| database | ||
|| object | 'select @@version_comment limit 1' ||
|| retcode | 0 ||
|| @ptr | CnYKOwo3MzkyMTM4Nzg5NzY0Oi9hd3MvcmRzL2NsdXN0ZXIvdGVzdC1hdXJvcmEtY2x1c3Rlci9hdWRpdBAFEjcaGAIF55NVCwAAAAF4+3PpAAXrU9EwAAABsiABKIO42p+fLjCZ9eOfny44lAFA7NEBSKNOUMZKEAYYAQ== ||
|| timestamp | 1588935969213615 ||
|| serverhost | test-aurora-master-instance ||
|| username | admin ||
|| host | 10.0.1.54 ||
|| connectionid| 137 ||
|| queryid | 14843 ||
|| operation | QUERY ||
|| database | ||
|| object | 'select now()' ||
|| retcode | 0 ||
|| @ptr | CnUKOwo3MzkyMTM4Nzg5NzY0Oi9hd3MvcmRzL2NsdXN0ZXIvdGVzdC1hdXJvcmEtY2x1c3Rlci9hdWRpdBABEjYaGAIF4+HZrwAAAAGPvrPYAAXrU9CgAAAF8iABKLeY2p+fLjDdwOOfny44eECLowFIgUVQp0EQBhgB ||
|| timestamp | 1588935969214210 ||
|| serverhost | test-aurora-master-instance ||
|| username | admin ||
|| host | 10.0.1.54 ||
|| connectionid| 137 ||
|| queryid | 14844 ||
|| operation | QUERY ||
|| database | ||
|| object | 'select user()' ||
|| retcode | 0 ||
|| @ptr | CnUKOwo3MzkyMTM4Nzg5NzY0Oi9hd3MvcmRzL2NsdXN0ZXIvdGVzdC1hdXJvcmEtY2x1c3Rlci9hdWRpdBABEjYaGAIF4+HZrwAAAAGPvrPYAAXrU9CgAAAF8iABKLeY2p+fLjDdwOOfny44eECLowFIgUVQp0EQCBgB ||
|| timestamp | 1588935969219838 ||
|| serverhost | test-aurora-master-instance ||
|| username | admin ||
|| host | 10.0.1.54 ||
|| connectionid| 137 ||
|| queryid | 14845 ||
|| operation | READ ||
|| database | mysql ||
|| object | user ||
|| retcode | ||
|| @ptr | CnUKOwo3MzkyMTM4Nzg5NzY0Oi9hd3MvcmRzL2NsdXN0ZXIvdGVzdC1hdXJvcmEtY2x1c3Rlci9hdWRpdBABEjYaGAIF4+HZrwAAAAGPvrPYAAXrU9CgAAAF8iABKLeY2p+fLjDdwOOfny44eECLowFIgUVQp0EQBxgB ||
|| timestamp | 1588935969219940 ||
|| serverhost | test-aurora-master-instance ||
|| username | admin ||
|| host | 10.0.1.54 ||
|| connectionid| 137 ||
|| queryid | 14845 ||
|| operation | QUERY ||
|| database | mysql ||
|| object | 'select host ||
|| retcode | user,authentication_string from mysql.user',0 ||
|| @ptr | CnYKOwo3MzkyMTM4Nzg5NzY0Oi9hd3MvcmRzL2NsdXN0ZXIvdGVzdC1hdXJvcmEtY2x1c3Rlci9hdWRpdBADEjcaGAIF6oyZCAAAAAB9A/cvAAXrU8rAAAAEgiABKNCa1J+fLjDN3t2fny44kgFAsLoBSIFKUKRGEGwYAQ== ||
|| timestamp | 1588935970114910 ||
|| serverhost | test-aurora-master-instance ||
|| username | admin ||
|| host | 10.0.1.54 ||
|| connectionid| 137 ||
|| queryid | 14852 ||
|| operation | QUERY ||
|| database | mysql ||
|| object | 'select now()' ||
|| retcode | 0 ||
|| @ptr | CnYKOwo3MzkyMTM4Nzg5NzY0Oi9hd3MvcmRzL2NsdXN0ZXIvdGVzdC1hdXJvcmEtY2x1c3Rlci9hdWRpdBACEjcaGAIF5gFPGAAAAAKWGiDIAAXrU86AAAAAgiABKM2D2J+fLjC5l+Gfny44gQFArbQBSPlIUJxFECEYAQ== ||
|+--------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+|
|| statistics ||
|+----------------------------------------------------------------------------------------------------------------+---------------------------------------------------------------------------+|
|| bytesScanned | 276868.0 ||
|| recordsMatched | 7.0 ||
|| recordsScanned | 1590.0 ||
|+----------------------------------------------------------------------------------------------------------------+---------------------------------------------------------------------------+|
sakamaki.kazuyoshi@HL00257
Atenaクエリ実行
Atenaにてテーブルを作成します。以下クエリを実行します。
CREATE EXTERNAL TABLE cwl_logs (
`timestamp` string,
`serverhost` string,
`username` string,
`host` string,
`connectionid` INTEGER,
`queryid` INTEGER,
`operation` string,
`database` string,
`object` string,
`retcode` string
)
PARTITIONED BY (year int, month int, day int ,hour int)
ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe'
WITH SERDEPROPERTIES (
'serialization.format' = ',',
'field.delim' = ','
) LOCATION 's3://cloudwatch-logs-for-athena-convert/test-aurora-cluster/audit/'
TBLPROPERTIES ('has_encrypted_data'='false');
監査ログの形式にあわせデータタイプを指定し、Kinesis Data Firehoseで指定したカスタムパーティションにあわせ、パーティションを作成しています。
パーティションを追加します。以下はクエリと実行結果です。
MSCK REPAIR TABLE cwl_logs
Partitions not in metastore: cwl_logs:year=2020/month=05/day=08/hour=11 Repair: Added partition to metastore cwl_logs:year=2020/month=05/day=08/hour=11
パーティションを確認します。以下、クエリと実行結果です。
SHOW PARTITIONS cwl_logs
year=2020/month=05/day=08/hour=11
さきほど実施したAuroraでのオペレーションを抽出してみます。以下、クエリと実行結果です。
SELECT * FROM cwl_logs WHERE year = 2020 AND month = 5 AND day = 8 AND hour = 11 AND username = 'admin';
"timestamp","serverhost","username","host","connectionid","queryid","operation","database","object","retcode","year","month","day","hour" "1588936254655024","test-aurora-master-instance","admin","10.0.1.54","137","0","DISCONNECT","","","0","2020","5","8","11" "1588935964038921","test-aurora-master-instance","admin","10.0.1.54","137","0","CONNECT","","","0","2020","5","8","11" "1588935964039296","test-aurora-master-instance","admin","10.0.1.54","137","14821","QUERY","","'select @@version_comment limit 1'","0","2020","5","8","11" "1588935969213615","test-aurora-master-instance","admin","10.0.1.54","137","14843","QUERY","","'select now()'","0","2020","5","8","11" "1588935969219838","test-aurora-master-instance","admin","10.0.1.54","137","14845","READ","mysql","user","","2020","5","8","11" "1588935969219940","test-aurora-master-instance","admin","10.0.1.54","137","14845","QUERY","mysql","'select host","user","2020","5","8","11" "1588935970114910","test-aurora-master-instance","admin","10.0.1.54","137","14852","QUERY","mysql","'select now()'","0","2020","5","8","11" "1588935969214210","test-aurora-master-instance","admin","10.0.1.54","137","14844","QUERY","","'select user()'","0","2020","5","8","11"
さきほど実施したAuroraでのオペレーションをログから確認することができました。
さいごに
確認したいオペレーションの日時などが把握できていれば、S3 Selectを利用することでお手軽に確認ができますが、一方、オペレーションの日時が不明な場合や、複数ファイルにまたがった確認を行う場合はAtenaを利用した方が捗るかと思います。ただし、作り込みが発生しますので、参照頻度などを考慮し、検討いただければと思います。
![[AWS Technical Support Note]ต้องระบุ endpoint url ด้วย DNS name อะไร หากต้องการเข้าถึง S3 โดยใช้ CLI ผ่าน S3 interface endpoint](https://devio2024-media.developers.io/image/upload/v1730273469/user-gen-eyecatch/uqmw2ndhu2v0te66pt7o.webp)
